Upload
edwin-marsh
View
215
Download
1
Tags:
Embed Size (px)
Citation preview
Measuring effectiveness
• Why should we measure?
• What should we measure?
• How should we go about it?
Food For Thought?
“it is difficult to come to any other conclusion except that the achievement of the objectives which have been set for the FSA are non-operational in the sense that no measurement of success can be achieved”
Professor Charles Goodhart, London School of Economics.
Measuring effectivness – WHY?
• Indicates progress against objectives• Tells you what is working, what is not • Informs strategic planning and resource allocation• Focuses on value for money with limited resources • Meets stakeholder expectations, as eventually some or
all of the following will demand proof of effectiveness:– Consumers– Industry– Government– Media
Effectivness – WHAT TO MEASURE?
Measure Inputs – are we doing things right?
Information resources• Publications• Website
Communication channels • Consumer Helpline• Information centre• Website• Advertising and media
Measure Outputs – arewe doing the right things?
• What is a ‘better informed consumer? Are Irish consumers better informed now than before?
• Do they know the costs, risks and benefits of the products available?
• Is the market safer and fairer than it was before?
Measuring effectivness – HOW ?
Inputs
• Examine the main
activities you undertake
• Decide the key performance
indicators
• Measure achievement
against these
Outputs
• More complex and difficult to
measure long term goals
• Do we influence consumer behaviour?
• How do we separate our influence
from other ‘influencers’ - TV, press
and other media, government,
consumer groups, lobbyists.
Measuring effectiveness - our activities
What we measured • Awareness levels
• Satisfaction with helpline
• Website evaluations
• Publications evaluations
Measuring Awareness of the Financial Regulator -overall increase in the year to
July 05 is up from 10% to 30%
Level of awareness
0%
5%
10%
15%
20%
25%
30%
35%
Quantitative assessment of the Lo-Call centre’s performance
• Satisfaction with the Lo-call line is very high• Callers are disproportionatly aged 35-64 years• ¾ of queries are claimed as resolved on first call• 9/10 will use again and/or recommended to a friend
“it wasn’t full of jargon, they didn’t use
terminology that is only used in the
financial services business”
“it was actually a joy because so many people now have such horrible telephone manners and
you're talking to machines and its press 1 for option this and then
you end up getting cut off so it was a great relief to be connected to a human
straight off”
Measuring effectivness of communication channels – website
• Qualitative research – focus groups & one-to-one accompanied surfs.
• Task-based assignments to assess the website in terms of:
• Design
• Content
• Functionality
• Exploration of comparative websites – Expectations vs. reality
Website Evaluation - Findings
“It is just bland. It just looks dull and
boring.”
“It is just a real black and white
website. There are no pictures, nothing visual to hold your
attention.”
Initial Impact• Very little ‘excitement’ about the website• Criticised for looking a bit ‘dull’• A number claimed it is a bit ‘boring looking’• Recognition that this is a ‘serious’ subject• Some argue serious doesn’t have to mean
boring• Calls for more pictures and images on the
site• Most want a bit more ‘life’
Other Websites
• The site ‘competes’ with other websites for ‘entertainment value’.
• It is easy to leave the site because ‘first look’ does not engage the reader.
Publications evaluation - findings
INSTITUTIONAL ‘feel’
• Little sense of ‘independence’
• Likely to be confused with bank brochure
• Old fashioned – ‘looks like school text book’
• “I thought they were trying to sell me a mortgage. I could imagine picking that up in the bank and it would really be just promoting their products”
“PICK ME UP” VALUE
• Nothing new at a first glance
• Dull use of colours, looks boring
• Few claimed they would be tempted to pick up
Measuring effectiveness - our long term goals
What are our goals?
• Better informed consumers
• Aware of costs, risks, benefits
• Safe and fair market
What do we expect to see if our goals are met?
Better informed consumers
What attitudes, skills and buyer behaviour might indicate consumers are ‘better informed?’
• they seek appropriate information and shop around before choosing a product or service;
• they know what products they need, at appropriate times in their lives
• they budget for things and plan ahead
• they are not usually persuaded to buy, but proactively choose products to suit their circumstances
• they know how to seek redress if things go wrong;
Some possibilities?
Awareness of costs, risks and benefits
What attitudes, skills and buyer behaviour might indicate consumers know more about the costs, risks and benefits?
• they know the relationship between risk and return
• they know the status of financial advisors and the implications of ‘tied’ vs. ‘independent’
• they know how to compare similar products
• they recognise when they need advice
• they ask questions, and don’t rely completely on the advice of any party
Some possibilities?
A safe and fair market
What indicators can we use to assess how safe or fair the market is?
• Product transparency – degree to which warnings, penalties etc. are signalled ‘up-front’ clearly and simply
• Degree of firms’ compliance with Codes of Conduct
• Sanctions imposed on non-compliant firms
• Competition – number of firms, product competition, price variation, degree of product differentiation, comparative advertising by firms.
Some possibilities
Measuring effectiveness - HOW
Longitudinal surveys – perhaps every five years
plus
Analysis of industry data
• Have to measure actual buyer behaviour as well as knowledge, attitudes and skills
• Must allow for fact that there is no ‘ideal’ of the informed consumer.
•Must allow for fact that consumers are not a homogenous group.
Methodology?
Issues?
Consumer Protection
Irish Financial Regulator in in the process of finalising a unified Code regulating the Conduct of Business of:
• Banks, • Insurance Companies, • Financial Intermediaries, • Investment Managers,• and Money-Lenders. The Code will also be extended to cover Stockbrokers and
Credit Unions.
Question: How can we ensure that effectiveness is a key criterion in the Code development process?
Ways of Considering Effectiveness of Codes
• Consultation with Industry,
• Regulatory Impact Analysis,
• International Best Practice.
Consultation with Industry
Benefits:• Knowledge of markets,• Interest in reputation of the Industry,• Knowledge of Impacts, particularly Costs.
Risks:• Regulatory Capture.
Regulatory Impact Analysis (RIA)
• Irish Government Requires All new Regulatory initiatives to be subjected to RIA.
• The Financial Regulator has just published its RIA in relation to the Code of Conduct : www.financialregulator.ie
• Key Elements: Costs and Benefits Impact on Competition Impact on Smaller Firms
Returning to Professor Goodhart:
• His conclusion was that one of the most valuable ways of considering the effectiveness of financial regulators was peer review.
• This supports the view that the Financial Consumer Education and Protection Forum is a very worthwhile initiative, and one that we should seek to exploit to its maximum potential.