Part 2/2 – Q&A With Ex User Experience Champion at LOVEFiLM
Monday, March 23rd, 2009This is the second part of the Q&A I have recently done with Craig Sullivan. The 1st part of the Q&A covered topics such as the LOVEFiLM user experience, ratings and reviews and their impact on conversion rates and comparing the user experience on some of the biggest high street retailers.
Paul Rouke:Like ourselves at PRWD, you are a big advocate of multi-variate testing (MVT), A/B testing and in-depth web analytics to measure and improve website performance. If you had 60 seconds to sell in multi-variate testing what would you say?
Good question. Here is the summary of thought:
- Your customers are the real experts about the design of your site, the landing pages, forms, copy, pictures, offers and calls-to-action
- Anyone who has an opinion at your workplace (including you) will be incapable of correctly guessing what works best for customers
- If you develop your website through guessing, you may get optimal results sometime shortly before the heat-death of the universe
- If you have any traffic, just use it for testing
MVT allows you to experiment by slicing up pages/emails/processes to use different versions of ‘stuff’ shown to customers. You might change the size, placement or colour of buttons or different graphics and copy text.
MVT takes your traffic and then shows people these variations to find which works best for your business goal. It’s like taking all your ideas, putting them in a barrel, stirring them around and seeing the best stuff bubble to the top.
There are side benefits too:
- You don’t have to agree a single design so all those annoying ideas that you end up arguing about in meetings? Test them instead!
- Since you don’t agree ‘one design’ those pesky meddling co-workers can still voice their opinion but as part of a controlled test
- You end up doing things that work for customers, not implementing 3 rounds of changes from your signoff chain
- The customer has spoken – now shut up. If someone called Tod from marketing really likes the blue button and the customer converts 10% better on the red button, guess we’ll go with that one then, eh Tod?
If you don’t do MVT, expect to continue spending time in meetings arguing about underlines, dropdowns, colours and copy text that nobody reads anyway. If your life is empty enough to need filling like this, MVT isn’t for you – knock yourself out.
This is the best technique for optimising conversion that I’ve come across, ever. I still blend this with UCD, usability testing and lots of other techniques as optimising broken stuff gets poorer results. A checkout process that sucks can only be optimised to suck a little bit less.
When it comes to experimenting with traffic, MVT saved my life and cancelled time-wasting meetings.
Paul Rouke: I have seen that you are currently reading Avinash Kaushik’s book Web Analytics – An Hour A Day. Avinash is also the Analytics evangelist for Google. How would you rate Google Analytics as a business class web analytics tool?
I think Avinash is one of the most inspirational authors on analytics, optimisation and insight. If you don’t read his stuff, go do it now.
I think Google Analytics has improved immensely and I know people using it quite happily for enterprise class analytics. However, this is because they are good analysts and business people, not just because of the tool.
I’ve seen too many analytics implementations fail abysmally, regardless of product or price. The main reasons are that:
- People haven’t done the business analysis to work out what ‘real’ reporting or KPI stuff is needed
- You don’t employ a part or full-time analyst ‘champion’
- Not enough time and effort is spent on implementation, data validation and training
You can’t just spend cash on a product and expect stuff to magically happen. People often switch around vendors hoping this will help and then make the same mistakes. It’s like buying expensive designer houseplants and forgetting you turned the last lot crispy within a week.
Paul Rouke: Which of the more advanced features of Google Analytics, such as personalised segments, funnels and goals do you personally use in your consultancy roles, and how have these insights helped to inform design decisions and convince key stakeholders of a need to improve?
I look at segments of customer traffic all the time. A particular interest of mine is looking at conversion funnel segments – where they come from and how they perform in A/B or multi-variate tests. I’ve used segmentation to optimise for a specific channel so this is shown only to those visitors. This means that you get better results than one set of creatives for general traffic.
Can you imagine a store where you walked in and it was all laid out with the offers and products you wanted, within easy reach and had a helpful sales person to give you a really good introduction?
Dynamic MVT allows you to shape-shift your page to fit the incoming traffic segment based on test data. Type a different keyword or phrase into google – you get a different landing page, a different experience and the site gets a higher conversion rate.
Paul Rouke: We are recently launched phase 1 of a user-centered designed internal business system, and I’ve presented on the subject of why this approach is vital, with a key business case being the age old adage of ‘time is money‘. Have you experienced internal business systems which would benefit from the likes of user testing and a more user-centric design approach, and have you actually being involved in projects for developing internal software and systems which have adopted a user-centric approach?
Internal business systems should be looked at for usability wins just like customer facing services. Intranets, Wiki systems, bespoke applications and particularly customer service back-ends can all show cost savings from training, faster processing and more effective ‘task conversion’.
Paul Rouke: Online usability within the financial sector can play a huge part in encouraging visitors to complete application processes, take out a loan and compare financial products. I notice you have also worked at PricewaterhouseCoopers. What have your experiences being within the financial sector and how well has user-centered design being embraced? Can you name any companies who are embracing a user-centric approach?
I wasn’t involved in UX work at PWC but I do come across a lot of financial sites that have particularly nasty forms that expect customers to jump through ‘hoops of fire’ to complete them.
I was helping two friends research and buy car insurance recently and was struck by how disjointed the process can be. The aggregators (who give comparisons) are improving a lot but are not perfect.
Many applications have to be finalized through the insurer’s site and these sometimes don’t copy the details or have different ways of asking questions. Some websites have forms that are bleeding *vast* amounts of potential cash, given the high cost of customer acquisition in this sector. A focus on analysis of bailout reasons, error rates on forms and general usability would save multiples of the budget required. A 1% lift in conversion through insight is probably worth decades of usability spend.
I used Morethan the other day, filled out a quote and took a phone call. When I finished talking I tried to complete it and it said ‘ your session has timed out’.
Since it hadn’t yet asked me my email address, how could I sign back in to complete it? I guess their reporting just has me down as some poor chum that wasn’t interested – the reverse is true. I appreciate security concerns over the data but how many of us complete this stuff at one sitting, or never get interrupted? In the RW (Real World), life is not single tasking. If you aren’t monitoring this stuff and fixing it, who’ll do it for you?
Paul Rouke: When developing new applications, software and e-commerce platforms, we combine user-centered design with an agile development methodology. Have you worked on projects that combine these 2 disciplines, and if you have, how did you find it?
Agile development methods aren’t new but they are ‘trendy’ with some companies. All I’ll say is that having stand up meetings does not an Agile process make
. UCD and Agile type methodologies go together extremely well because rapid user testing and prototyping allows you to fit with an iterative release strategy (small, nimble releases not large monolithic developments). This also blends well with optimisation work because you can prioritise stuff that has a large customer impact or bottom line improvement and release this incrementally. If your site loses 7% of form fillers because of a couple of errors, you need to get these fixed and live quickly.
My advice to people that focus on Agile is to think of a Venn diagram where there are ‘Experienced Project Managers’ in one circle and ‘Agile approach’ in another. The intersection of the two is the place to look at but I’d rather have experience and cross train someone with Agile. Exposure to these methods or training courses doesn’t make someone good at the core skills of project management.
Paul Rouke: Who do you rate as you your biggest inspirations in the world of usability and user experience?
Hard to list all of them:
- Don Norman, Jakob Nielsen and Jared Spool for inspiring me generally
- Alan Cooper for ‘The Inmates are running the asylum’
- Seth Godin for ‘Permission Marketing’
- Steve Krug for the most usable usability book, ‘Don’t make me think’
- Jim Sterne, Eric Peterson, Hurol Inan and Avinash Kaushik for Web Analytics ‘training’ and being ‘Data Junkies’
- Hurol Inan for ‘Search Analytics’
- Luke Wroblowski and 37 Signals for their work on forms usability.
Paul Rouke: I am a huge advocate of multi-faceted navigation, especially for sites with high levels of content. What have been your experiences of this type of advanced navigation, and which sectors has been most pro-active in giving their visitors the power that comes with this advanced, personalised way of navigating large product and content sets?
My experience might be different to yours but I’ve seen a lot of implementations with electrical goods online. If you are looking for a stainless steel fridge where the door fits on the left hinge, with a certain height, colour and price – faceted filtering is ideal.
How to present large result sets and allow a visitor to flip the data, shrink it, expand it and reset it is one of the biggest challenges I’ve ever had. Faceted filtering software and services are very good (there are even open source projects like SOLR now). However, you need to ensure the interface presented doesn’t undo the nice data handling underneath. A library without an index is just an unusable place and so are faceted filtering systems without a good UI.

