Using Skype for Moderated Remote Usability Testing

By | Research, Uncategorized, Usability Testing, UX | 7 Comments

I used Skype and Ecamms Call Recorder for Skype to conduct moderated remote usability testing around the world on a clickable Marvel app prototype. Heres how I got on.

I’ve been working for a Japanese sportswear company who wanted some usability testing done on a new feature they were planning for their fitness app. As the app has a healthy (pun intended) global user base they were keen to test with users from around the word which mean’t remote usability testing on a clickable prototype put together in Marvel app.

When I’ve conducted moderated remote usability testing in the past I’ve used screen sharing services like Webex and GoToMeeting but I’ve always found the requirement on the end user to install various plugins troublesome and the cost (at the time) was prohibitive.

My loose set of requirements this time around was:

  • No plugins to install
  • Low cost
  • Reliable
  • Record screen, audio and video of the participant

I settled on Skype in the end as its pretty ubiquitous these days and although the app would need setting up and installing the chances were that a lot of respondents would already have it up and running.  Because I had a few thousand potential participants I took a gamble and was explicit in the screener recruitment survey that we would be using and Skype on a desktop or laptop and this was a requirement. Despite this I still got a 20% response rate which was better than I normally get for face to face interviews (what also helped was a fairly loose set of requirements for participants and also doing the tests over the weekend).

Skype doesn’t have an inbuilt recording capability but there are a few plugins that can allow this. I used Call Recorder for Skype by Ecamm and at $29.95 was pleased with the results.

What went well with the testing:

  •  Low barrier to entry mean’t that I got a good response and including Skype as a requirement didn’t seem to put people off
  •  Not having to mess about with talking people through installing plugins on the day took a lot of the pressure off me and allowed me to focus on the testing
  •  The call and video quality was excellent so I was able to include excerpts in the final presentation deck for the client
  •  Conducting the testing at the weekend meant I got a good response and because it was remote meant I could spread it out over a couple of days without having to hire meeting rooms
  •  People seemed more at ease with the test and opened up a lot quicker because they were at home
  • It was cheap to run compared with hiring meeting room space

What went less well:

  •  Skype wasn’t quite as flawless as I hoped. Two out of the five couldn’t share their screen without Skype crashing.
  •  I also had the call drop a few times for no particular reason which was annoying and interrupted the flow

So would I use Skype again? Probably yes. Despite its flaws it was pretty easy to use for the participants and although its no substitute for face to face interviews it was still pretty effective and I got lots of good feedback. Conducting it over the weekend wasn’t ideal (for me!) but mean’t I could talk to more people and turn around the test results quickly.

5 years on … Lessons learnt from my time in the freelance UX trenches

By | Freelance, Portfolio, UX | 2 Comments

Five years ago this month I quit the comfort of my full time job and went freelance. I often get asked  about how to go about becoming freelance so I thought I’d mark this anniversary by listing a few things I’ve learn’t upon the way (in no particular order):

1. If you’re going to go freelance do it properly

When I left my full time job I spent the first 6 months not really sure whether I wanted to go perm or go freelance. The result was a bit of a mess of me half applying for jobs and half committing to the freelance life. In hindsight I should have focused more on the freelance thing and set myself a deadline where I could reassess my career.

Remember you’ll also need a few months money to live on while you wait for your first invoices to be paid. I had some inheritance so I gave myself 3 months to make a go of it. My plan was that If after the first two months It all went horribly wrong I’d still have a month to find a permanent job. I still have that money in the bank.

2. Find a mentor

I’m lucky that I know a lot of freelancers. When I first started out I found it really useful to meet up with them from time to time and pick their brains about freelance life and what I should be doing. Its worth getting to know some freelancers (doesn’t matter what line of work they are in) just to sense check your decisions and sound them out for advice.

3. Get a website 

I’ve had a crappy website for years. This year I spent a bit of downtime at the beginning of the year updating it to something only slightly less embarrassing. The result of which is that whereas before I didn’t get very many work enquires from it I now get 1 a day on average.  Wish I’d done it years ago. I bought a WordPress theme and hacked it about to meet my needs but you are probably much better at coding than me. Have a look at other freelancers websites for inspiration.

4. Get a (meaningful) portfolio

There have been plenty of people blog recently about the value of UX portfolios but I can talk from experience of having been on both sides of the hiring table that it is worth spending time on one. Just remember that nobody is interested in a list of projects with shiny screengrabs and Photoshop mockups. What hiring people want to read is a story about the projects you worked on starting with the objectives, what you did on the project and the outcomes. (I’ve got a longer blog post drafted on this which I’ll save for another time)

5. Use LinkedIn

I have a love hate relationship with LinkedIn but the fact is that I get a lot of work out of it so spend the time honing your profile. Think of the kind of skills people will be looking for an get those key words in your profile. Also keep the copy tight and to the point and use bullets to highlight key skills.

6. Network Network Network

Get along to as many networking events as you can. The UX (and web industry generally) is a friendly space to be in so get to know people as you can. I quite often pass work onto people I know if I can’t take it on and the reverse has happened to me as well. It also doesn’t hurt to do a few talks and get a bit of a name for yourself if you can.


Some months are lean some months you’ve got work coming out of your ears. You can’t predict how much work you have so don’t bother trying. The only things I noticed is that from mid December to mid January things are quiet and also things quiet down a bit in August. Also remember that you are not a machine and take time off. Work will always be there when you get back.

8. No job is ever guaranteed

I’ve learn’t this the hard way but it doesn’t matter how many contracts you’ve signed no work is guaranteed so be prepared. I’ve had projects stop a couple of days before they were meant to begin and half way through (nothing to do with me!). Unfortunately it comes with the territory so make sure you’ve always got a couple of months money in the bank just in case it happens to you.

9. Keep track of your finances

Its much easier to sort out invoicing, expenses and tax as you go along rather than leave it to the last minute. I try and keep a 1-2 of hours a week to one side just to keep up to date on admin. There are lots of finance packages out there but I recommend Freeagent (heres a referral code if you fancy giving it a try for a 10% discount – 33n0dtl2). I know some people use spreadsheets but for the money its worth having something like Freeagent to streamline the process and leave you a bit more time for the more important things in life.

10. Set your rates

Don’t ever try and be the cheapest. Sell yourself on producing quality work and set your rates accordingly. I have a sliding scale of rates depending on who the client is, what the job is, where its based and how long its for. Having said that I tend not to worry about money too much. I do this job because I genuinely love it and would rather do an interesting project than work on something well paid. If your main motivation for becoming freelance is earning loads then there is plenty of well paid work in the finance and gambling areas.

11. Get your game face on 

You’ll be expected to hit the ground running when you start a new project and the people in your team will be looking to you for the answers. I rarely get to work with other UX people (when I do its a treat) so nobody will be there to cover your back. If you are serious about becoming freelance make sure you know your trade inside and out. There will be lots of occasions where you will need to learn as you go along but when you do this make sure you’re not making too much of a leap into the unknown as it will come back to bite you.

12. Dont be the Flash guy

Remember Flash? Don’t be that person who still only does that one thing. Try and keep ahead of the game and keep your skills sharp by reading blogs, following the right people on Twitter and going along to talks and conferences.

13. Document and record everything

I generate a lot of paperwork and sketches. I try and keep a record of everything I do for my portfolio and to evidence my work in case anyone ever asks why certain decisions have been made. I don’t keep hard copies but I photograph everything and file it away. You never know when it will be useful.

14. Find your preferred way of working

Theres no right way of working or type of project, its whatever suits you. Some UX freelancers prefer to work on retainers or will only work from home. I prefer working on bigger projects full time and generally work onsite. Its just whatever works for you.

So there you go, a few things I’ve learnt along the way. I hope if you are planning on making the switch you found this useful. At the moment there is still plenty of work to go around (especially for experienced people or those with specialisms) but the market is becoming a little crowded as more people make the switch. Fortunately as the industry becomes more established so does the quantity of work available so its still a viable option.

There are probably lots of things I’ve forgotten so feel free to ask and buy me a beer sometime and I’ll tell you all the sordid secrets!

Good luck!!

How to use a user centred approach to prioritising features in your site

By | Agile, UX | One Comment

As part of my work at the BFI I’ve been thinking about how best we can integrate a more user centred approach to the prioritisation of the user stories in the project backlog, after all we’re designing the site for the user not us so why shouldn’t they have a say in what features and pages get built?

One solution to this problem is to use the Kano Model of Customer Satisfaction which was developed by Professor Noriaki Kano in the 1980’s. In a nutshell Kano says that user preferences for functions or features can be distilled into 5 categories:

1) Attractive Quality – The ‘nice to haves’  or delighters, features and functions which add to the experience but are not essential and wouldn’t be missed if not included. For example the little bag of sweets that Firebox include in their parcels.

2) One Dimensional – features which (when they work and are done well)  add a great deal but cause frustration when they do not deliver as promised.

3) Must Be – The givens. These are the things we expect to be provided and while they do not add to the users satisfaction cause a great deal of frustration when missing or badly implemented.

4) Indifferent – Features which are neither good or bad and do not result in either satisfaction or dissatisfaction with users.

5 Reverse   – The best way to describe this is assuming that all your users are alike. For example a gadget which is so overloaded with features it is at the expense of the user who thinks your product is overly complicated.

Kano illustrates this best in this diagram:

Kano Model

Now using the Kano Model in user centred design is well known, in fact Andrew Harder who gave an excellent talk on Kano as part of his User Research as a generative partner with design talk at this years UX People which is what inspired me to look into this further.

We will be using a variation of the model to test users appetite for the features we are proposing as part of our usability testing sessions. We conduct one round of user testing towards the end of each sprint (I’ll talk about our approach to user testing in another post). At the end of the testing session we will ask users to rate the features of the wireframes  (a feature could be a page, a section or a widget) and use an aggregate of those ratings as a guide for when the Product Owner is prioritising User Stories (features) in the Project Backlog.

The rating we will use will look something like this:

How would you feel if feature x was to be in the new site?

  • I’d really like it
  • I’d expect it
  • I might use it
  • Its unlikely I’ll ever use it
  • I really dislike it

These responses are then aggregated according to the persona (we recruit users for testing against personas, I’ll talk further about how we use personas in another post). So for example if we recruited 5 people who matched our Nick persona and 3 of them responded very positively to feature X then the corresponding user story in the backlog might read. “Nick really likes feature X and would like to see it included” or if we didn’t have a majority the story could read ” On the whole Nick really likes feature X and would like to see it in the new site”

So there you go. This model isn’t intended to replace the Product Owners backlog prioritisation function but should help act as a sanity check when looking at the user stories.

Looking for UX design and research help with your project? Get in touch