Crowdsourcing for Customer Experience Management Success

Steven W Beauchem http://www.linkedin.com/in/stevenbeauchem

http://yetanothervaliantattempt.blogspot.com/ Phone: 847-224-8346

E-Mail: [email protected]

Crowdsourcing for Customer

Experience Management

Success Fall 2012

Why Read This Report As the importance of effective digital customer experience continues to grow, poor content quality will become a major limiter in digital business success. Interactive Marketing Technology (IMT) professionals must expand their focus beyond the technology platform, and provide solutions that improve the impact of content to the business. Recent developments in crowdsourcing, specifically in the area of cloud labor, provide unique opportunities for IMT pros to take a proactive approach to content quality, both during content migration and after the Customer Experience Management (CXM) platform is operational.

Crowdsourcing for Customer Experience Management Success

1

Report Highlights

An Effective Customer Experience Solution Requires Content Quality

Customer Experience Management (CXM) platforms have evolved well beyond managing html, images and keywords. A dynamic, personalized customer experience depends on content that is wrapped with rich metadata, structured for publishing across multiple channels, and assigned to relevant business categories. As customer experience capabilities expand, the amount of effort required to maintain content quality grows in parallel.

Cloud Labor is Emerging as a Solution to Maintain Content Quality

Content producers are overstretched trying to keep up with new content, and they simply lack the bandwidth to keep up with ongoing management. By leveraging a globally distributed workforce of task-based workers (i.e., Cloud Labor) to offload content quality maintenance, content producers can focus on building and launching new content while crowd workers keep existing content in excellent condition.

IMT is the Architect of Cloud Labor Integration

In order for cloud labor solutions to function effectively, IMT pros need to collaborate from the start with content producers to identify best uses for cloud labor resources, design cloud labor processes and work activities, and develop the integration points between the CXM and cloud labor platforms.


2

Firms Scramble to Support Rich, Meaningful Customer Experiences As leading edge companies have demonstrated the business value of focusing on customer experience quality, so too have customers become accustomed to digital touchpoints that are easy, useful and enjoyablei. Personalization, for example, makes or breaks customer experiences, and winning companies make this a key value, rather than just a feature.”ii This resulting population of increasingly savvy customers has left many firms struggling to keep up, both with their direct competitors and with expectations being created by digital innovators (i.e., the “Amazon Effect”). And, as business-to-business (B2B) firms are starting to learn, customer experience is not just a consumer play; effective customer experience has the potential to drive measurable business benefit regardless of audienceiii.

Interactive marketing technology (IMT) professionals need to play a critical role in their firms’ customer experience strategies. As experiences grow in sophistication, so does the level of pre and post-delivery content support required to realize the value of the delivered solution. Companies build rich, highly engaging customer experiences by combining customer experience management (CXM) solutions and the content that populates them. In order for CXM features (e.g., search, personalization, analytics, marketing automation, commerce) to function as advertised, content quality (Figure 1) is a must.

FIGURE 1: ATTRIBUTES OF CONTENT QUALITY—INHERENT VALUE, STRUCTURE, METADATA, AND TAXONOMY

What’s frustrating to many IMT professionals is that they are fully aware of the content quality issues gumming up their solutions, and it’s not pretty. Firms port content into their shiny “next gen” CXM solution from multiple publishing systems, each with its own metadata structures, quality standards and levels of quality enforcement. As a result, dynamic features and personalization fall prey to inconsistent, suboptimal content quality. Or, project time and cost increases exponentially as each additional content store is manually audited and optimizediv. And, like the second law of thermodynamics, entropy starts kicking in the moment the CXM solution is launched; unless there have been dramatic cultural and process changes in the content producing organization, content quality will continue to degrade.

IMT Professionals Deal With the Content Quality Mess It’s easy to point fingers when it comes to issues of content quality. IMT professionals blame content producers, who in turn complain about hard-to-use content management systems, and everyone takes a shot at creative agencies. In reality, the situation is much more complex, emerging from a number of prior decisions and constraints into the hot mess we have today:

Inherent Value

Structure Metadata

Taxonomy

Is the content accurate, useful and utilized?

Is the content structure consistently abstracted from presentation to support ease of reuse?

Are the content descriptive attributes complete and accurately populated?

Has the content been categorized appropriately

to support customer experience features?


3

Interactive budgets trend upwards, but interactive teams remain under-resourced

The effort required to effectively optimize content for effective customer experience is frequently beyond the capacity of most interactive teams to effectively managev. Companies have attempted a variety of “magic bullet” solutions (e.g., auto-classification, dynamic transformation, syndication) to reduce workload, but with mixed results. And, with every minor tweak to content architecture, legacy content recedes further away from optimal. Ideal or not, content teams tend to ignore yesterday’s issues to focus on the burning platform.

Indiscriminate use of outsourcing to manage content exacerbate the issue

The quality of vendors managing content varies widely, as does the ability of interactive teams to define and manage vendor relationships. This creates challenges ranging from misunderstandings regarding content requirements to project delays to unacceptable deliverablesvi. Vendor agreements also tend to focus on specific quality attributes (e.g., browser compatibility), with little attention paid to elements that drive a personalized customer experience (e.g., metadata, structure, taxonomy). With the firm’s interactive team focused on getting content deployed, they may limit or skip content optimization activities altogether.

Interactive teams swim in a torrent of user-generated content

On top of all of this, interactive teams have new mandates to: encourage the creation of user-generated content (UGC); screen the volumes of assets being generated; and, identify items with the potential for extended use. Once acquired, UGC has to go through the same content optimization processesvii as owned content, and we already know how likely those are to be completed.

Management of content as a strategic asset has been lacking

Let’s face it: firms just aren’t very good at managing content. Instead, they focus on short-term gains: investing in visual appeal at the expense of content architecture; gold-plating initial releases of content management solutions because of funding uncertainties for follow-on work; and avoiding the perceived drudgery of content maintenance and optimization in favor of deploying new content. While the rapid pace of change in content management technologies will continue to create optimization challenges, solution owners haven’t done themselves any favors by focusing on immediate needs at the expense of future possibilities.

Cloud Labor: Now Ready to Support Customer Experiences Fortunately, recent trends in crowdsourcing create potential opportunities to systematically and cost-effectively improve existing content quality and optimize the quality of new content as it is deployed. Cloud labor is different from outsourcing:

“Cloud Labor is the leveraging of a distributed virtual labor pool, available on-demand to fulfill a range of tasks from simple to complex. Crowdsourcing is used to connect labor demand and supply. Virtual workers perform activities that range from simple to specialized tasks.”viii

One cloud worker working for a client recently sent this email to the document author, apologizing for errors in her completed assignments: “Hi – I'm really sorry – I completed 2 of your tasks before I realized that they were the same. I had sorted differently and yours appeared twice for me. I am really sorry and I don't expect to be paid.” While cloud labor changes the nature of the relationship between workers and managers, individuals generally take their cloud labor responsibilities as seriously as they would full-time employment.

Over the past 24 months, cloud labor has emerged as a viable option for getting work done within the enterprise. Organizations are still exploring potential uses for a skilled, on-demand workforce to complete activities requiring too much human judgment to automate, yet insufficiently valuable to cost-justify dedicated resources. However, it is clear from cloud labor industry growth (75% in 2011 to $375 million) and the arrival of enterprise early adopters (e.g., Amazonix, Target Stores, Microsoft, eBay) that firms are responding positively to the cloud labor value propositionx.


4

In comparison to outsourcing, cloud labor provides greater flexibility at lower cost; and, when well designed and managed, can achieve and exceed outsourcing quality levels. The structural differences between the two extended workforce models are the source of crowdsourcing’s relative benefits (Figure 2).

FIGURE 2: CLOUD LABOR VS OUTSOURCING

Cloud Labor Outsourcing

Unit of Work Work is managed and compensated at the task level. As a result, workers are rewarded for completing more tasks per time period.

Work is generally managed by task, but compensated on an hourly basis. Unless specified in the outsourcing agreement, workers do not have an incentive to increase their task output.

Business Relationship Workers function as independent contractors, and generally self-select tasks for which they are qualified and which hold interestxi.

Workers are employees of a contracting organization. To meet delivery commitments, workers may be assigned to activities for which they are not qualified.

Vendor Management Simple: since work is managed at the task level, work can be screened and approved / rejected as it is completed. Rejected tasks are not compensated. Task design can be tested, revised and validated in small batches using agile principles.

Complex: quality standards must be clearly defined in a legal agreement and managed with the outsourcing vendor. Issues in delivery quality are often not identified until a major review milestone. Rejected work will typically result in additional costs for rework. Firms have limited input into work design beyond requirements definition.

Scalability Dynamic: with some platforms hosting +100k cloud workers, new projects can get moving quickly. Note that projects requiring specific domain expertise or up-front learning and certification will take longer to staff.

Flexible: new projects require time for vendor identification, contract negotiation and approval. Additional time is often required to scale the project team to production levels.

What Types Of Content Management Activities Are Well Suited To Cloud Labor?

When considering what content management activities to support with cloud labor, organizations should start with the list of content maintenance tasks that keep getting delayed into next week/month/quarter (see Figure 3). While cloud labor solutions will evolve to address more complex activities, you can achieve consistent results today given that your cloud labor assignments meet the following criteria:

• The tasks require human judgment and intuition. While automated solutions such as Autonomy’s MetaTagger provide effective textual analysis and content classification, results vary based on input quality. An experienced cloud worker can be more effective than a technology-based solution at judging relevance and classifying content, especially where images, audio or video are involved, or where there is limited content of high quality to “train” a


5

technology solutionxii. Activities requiring technical knowledge (e.g., classifying content on medical conditions for a healthcare website) will require additional time to identify appropriate workers and/or provide additional worker training.

• The tasks are discrete. a cloud labor task needs to be a self-contained activity, with clearly defined inputs and outputs. A worker should be able to complete a selected task without external coordination or assistance (e.g., calling a support desk for clarification or coordinating with upstream / downstream participants in a process). Cloud laborers generally seek to compress as much work into as little time as possible, either because they’re seeking to generate as much revenue as possible; or, because it’s a secondary source of income and they have limited time in which to complete cloud labor tasks. In addition, cloud labor platforms aren’t sufficiently mature to support multiple workers collaborating within a single activity.

• Task fulfillment cost is lower than equivalent internal or outsourced resources. As firms gain experience with the cloud labor approach, it will become fairly straightforward to build cost models for cloud labor tasks. Performing detailed cost analysis will help to justify investments in development and infrastructure required to support ongoing leverage of cloud labor.

FIGURE 3: EXAMPLE CONTENT MANAGEMENT ACTIVITIES SUITED FOR CLOUD LABOR

Task Description Vendors

Tagging / Categorization Selection of categories or taxonomy attributes relevant to a content item (text or media). Note that cloud labor is currently less effective for highly technical vocabularies.

Amazon, CloudFactory, CrowdFlower, CrowdSource, TagCow (images), Tagasauris (images)

Transcription Creation of transcripts from audio or video sources for use with search engine optimization (SEO) or ADA compliance.

CastingWords, Speechpad

Content Descriptions Creation of teaser content, product descriptions or other summary content to support alternate display formats (e.g., list views, syndication).

Servio, CrowdFlower, CrowdSource

Content Moderation Review of new posts to communities, product forums, review sites, etc., to ensure suitability of content, compliance with behavior policies.

CrowdSource, CrowdFlower

Relevance Evaluation Determination of whether a given description or attribute is relevant to the content asset to which it is assigned (e.g., “is this a picture of a Ford Explorer?”).

Amazon, CloudFactory, CrowdFlower, CrowdSource

Content Extraction Scraping html from an existing URL and transferring into a structured content template.

Amazon

Translation Translation of content from source language into one or more target languages. Note that cloud labor is

Smartling, Amara


6

Task Description Vendors

currently less effective for highly technical vocabularies, and may require additional review.

Usability Testing Feedback from site or application testers on design patterns, navigation, feature validation, etc. Primarily useful for generic usability testing and quality assurance. As vendors expand and refine their crowd labor segmentation, it should be possible to recruit cloud workers specific to a firm’s target audience.

TryMyUI

Key Considerations for Planning Your Cloud Labor Solution Organizations can group content management processes c into two categories: batch processes (e.g., content extraction to support a web content management (WCM) migration, or bulk update of content with new metadata attributes) and workflow processes (e.g., review of a new customer forum post for appropriateness, or confirmation that a personalization feature is displaying the correct content). While cloud labor is effective for both process types, different approaches are required.

For batch processes (especially those with limited reuse), organizations should design with a primary focus on efficiency, i.e., building the minimum solution that accomplishes the desired results. In particular, they should:

• Develop worker user experience leveraging “out-of-the-box” platform capabilities. Most platforms provide basic task design and editing tools, as well as templates for common tasks. Leverage these to get up and running quickly.

• Invest lightly in worker training and certification resources, assuming limited reuse. Unless there is significant detail or nuance you need to communicate to your cloud workers, stick with the basics (html, screen captures, screencasts, etc.).

• Invest in pilot processes. Organizations should compare results from pilots to a pre-defined valid response set to troubleshoot worker training and task design issues. Look to the cloud worker community for feedback on issues and opportunities for improvement.

• Bulk load task results to content repository via scripting. Most cloud labor platforms are capable of exporting results in CSV and other data formats.

• Consider SaaS solutions that are specific to the batch process. The vendors identified in Figure 3 have already done a lot of the heavy lifting to get you moving quickly on certain content management processes.

For workflow processes, firms should build for effectiveness, with the primary objective of delivering consistent results over time. In particular, they should:

• Design and develop reusable worker user experience leveraging the firm’s CXM stack. The CXM toolset should be capable of providing a much more effective user experience than the crowd labor platform. Given that the crowd labor output would end up here anyway, it makes sense to build here in the first place.

• Explore opportunities to encapsulate external tasks within CM workflow processes. Most content management vendors provide capabilities to externalize workflow activities (generally via structured XML output and input). This creates opportunities to initiate crowd labor tasks based on events within the CM solution (e.g., a new product description is created).


7

• Integrate to the cloud labor platform for specific worker and task management services. The cloud labor vendor

should provide web services APIs for the capabilities a workflow solution will require (e.g., authentication, authorization, worker task selection, task approval).

• Consider third-party solutions for ongoing workforce management. A vendor ecosystem is emerging to manage worker qualifications and maintain consistent result quality over time. This adds to the ongoing solution cost, but reduces up-front investment in quality control expertise.

• Invest in robust worker training and certification. Given the need for consistent result quality and repeatability over time, targeted investments in richer training assets (e.g., modular, just-in-time training) are justified.

Recommendations

Leverage Cloud Labor for Content Quality Improvements While cloud labor is still climbing the maturity curve, organizations can recognize clear value in using it to address specific quality issues with customer experience-based content. It’s likely that your content professionals are having similar thoughts, and a coordinated approach to piloting and implementing crowd labor solutions will help to establish good practices for future expansion of cloud labor applications. Specifically, IMT professionals need to:

• Understand the role of compensation in cloud labor performance. One of the biggest challenges firms face when implementing cloud labor solutions is figuring out task pricing. Recent research has indicated that task pricing has a limited impact on result quality—more compensation doesn’t buy better outcomes. However, higher task prices do result in faster completion; workers tend to chase better payouts and will complete as many as they can.xiii Explore how other organizations are pricing similar tasks, and adjust based on your priority (i.e., speed vs. cost).

• Focus on user experience. Firms getting started in the use of cloud labor often assume they can shortchange experience design—that they can “throw something together”. Effective task design is the primary factor in achieving result quality from crowd labor, and poor task design will result in additional time and expense to get to desired results. Engage design resources and user test your tasks with crowd workers to ensure that the objectives of the task are easy to understand and efficient to complete.

• Clearly define accountability for cloud worker management. It’s easy for stakeholders to forget that someone has to be responsible for care and feeding of the cloud workforce. Your author recommends that content producers maintain responsibility for cloud workforce management, given that the role has a direct impact on quality of content. Regardless of who owns the activity, adequate resources and funding must be allocated.

• Get paid to explore the possibilities. One of the best ways to get a better understanding of cloud labor is to spend a few hours as a cloud worker. The experience will help to ground you regarding what works (or doesn’t) in task design and user experience, as well as the type of attention and discipline required to be successful as a cloud worker. And, you can always use your earnings to buy yourself lunch (or coffee, depending on how successful you are).


8

Endnotes i Burns, Megan. "The Business Impact Of Customer Experience, 2011." Forrester.com. Forrester Research, 07 July 2011. Web. 06 June 2012.

<http://www.forrester.com/The Business Impact Of Customer Experience 2011/-/E-RES59072?objectid=RES59072>. ii Brave, Scott. "Personalization Is Not A Feature." TechCrunch. TechCrunch, 18 May 2012. Web. 06 June 2012.

<http://techcrunch.com/2012/05/18/personalization-is-not-a-feature/>. iii Hagen, Paul. "How B2B Firms Improve Customer and Partner Experience." Forrester.com. Forrester Research, 14 Mar. 2012. Web. 06 June 2012.

<http://www.forrester.com/home#/How+B2B+Firms+Improve+Customer+And+Partner+Experience/quickscan/-/E-RES60388>. iv For the purposes of this document, “content optimization” is the collection of content provider activities performed to ensure content quality.

These activities include: editorial review, search engine optimization, metadata and taxonomy population, presentation testing in target environments, etc.

v VanBoskirk, Shar. "The Future of Interactive Marketing." Forrester.com. Forrester Research, 04 Apr. 2011. Web. 06 June 2012. <http://www.forrester.com/home#/The+Future+Of+Interactive+Marketing/fulltext/-/E-RES59137>.

vi The Current State of Digital Content. Rep. ValueNotes, Mar. 2011. Web. 6 June 2012. <http://www.artisarkapps.com/admin/s4carlisle/wp-content/uploads/2012/01/Market-Study-Report.pdf>.

vii For the purposes of this document, “content optimization” is the collection of content provider activities performed to ensure content quality. These activities include: editorial review, search engine optimization, metadata and taxonomy population, presentation testing in target environments, etc.

viii "Cloud Labor." www.crowdsourcing.org. Crowdsourcing LLC, 06 June 2012. Web. 06 June 2012. <http://www.crowdsourcing.org/community/cloud-labor/6>.

ix While a number of cloud labor platform providers are in the process of establishing themselves, the vendor that continues to receive the most attention is Amazon, with their Mechanical Turk (AMT). Initially built to support Amazon’s own needs for content maintenance, AMT now offers a robust, extensible cloud labor Platform as a Service (PaaS) that can be leveraged across a wide range of use cases. As a result, multiple solution providers (including most of the vendors identified in Figure 3) leverage the AMT services framework to deliver some portion of their feature set . For IMT professionals seeking a low-barrier means to explore cloud labor and perform pilot testing, AMT offers a fully-featured sandbox environment to build and deploy worker tasks.

x "Enterprise Crowdsourcing Research Report." Www.crowdsourcing.org. Crowdsourcing LLC, 10 Mar. 2012. Web. 06 June 2012. <http://www.crowdsourcing.org/editorial/enterprise-crowdsourcing-research-report-by-massolution/11736>.

xi A significant factor constraining more rapid adoption is the lack of clarity around the employment status of cloud laborers. For more information, see: Wolfson, Stephen, and Matthew Lease. "Look before You Leap: Legal Pitfalls of Crowdsourcing." Wiley Online Library. John Wiley & Sons, Inc., 11 Jan. 2012. Web. 06 June 2012. <http://onlinelibrary.wiley.com/doi/10.1002/meet.2011.14504801135/abstract>.

xii In November 2012, the National Institute of Standards and Technology will be holding its 21st annual Text Retrieval Conference (TREC). For the last two (2) years, the conference has included a Crowdsourcing track whose focus is on leveraging human relevance judgments to inform / train automated classification solutions. For more information see: http://trec.nist.gov/.

xiii Ipeirotis, Panos, and Praveen Paritosh. "Managing Crowdsourced Human Computation." Research at Google. Google, 2011. Web. 06 June 2012. <http://research.google.com/pubs/pub36946.html>.