Business IntelligenceHow to translate different measurements and scores into

Business Intelligence

Tom Shaw (Capita TI)Paul Mangell (Alpha CRC)Scott Carothers (Kinetic TheTechnologyAgency)

The Goal – Improve our Industry using Business Intelligence

?

What Data? How to capture?

What’s relevant? To whom?

How to interpret?

Adequacy

Fluency

TM Leveraging

User Sentiment

MT Leveraging

Accuracy

Style / Readability

Translator Quality

User Engagement

TER, Bleu, Meteor

Per Word

Per Hour CTR

PE Distance

TAT

Translation Quality – Business Intelligence from a LSP perspective

Tier 3 content

Tier 2 content

Tier 1 content

Support content, non-publishable user driven or reference

Publishable content for user assistance or product support related

Highly visible content, such as Marketing or software localization as

well as legal

Quality Evaluation Triangle

QUALITY

Users

LSPBuyer

The user experience should be equal or better of that achieved in Source language

Customising metrics and QA process in coordination with the

buyer will assure expectations are measured adequately

The buyer’s “internal clients” play a key role in setting up quality

expectations and measuring success

Quality Evaluation - LSP and Buyer

Content type specific QA process and metricsLSP Evaluation Criteria / Model:

Error Typology / Adequacy / Fluency

Territory specific requirements and preferences

Buyer Internal Clients

Quality Evaluation - Users

Users User feedback and content engagement levels

Evaluation Criteria / Model:User Activity / Engagement

Quality Evaluation Models - Thoughts

1) Quality Evaluation Models have to be closely linked to purpose / end use

2) Standard models not always good enough. Customised to specific needs

3) Metrics / Benchmarks should be set with end user in mind

……………………. Who should drive? LSP<>Buyer, Buyer<>Users….

Common Metrics the Buyers Maintain

Most common metrics:

• Which LSP has the assignment

• Delivery Time

• How long did post-editing take?

Most Valuable Metrics for the Buyer

1. Aggregate volume

2. Turnaround time

3. TM Analysis

4. LSP ratings

5. Cost tracking

Enterprise Metrics Dashboard

Checks and balances in collecting and sharing data

• How will the data be collected, analysed, presented?

• Is all data good data?

• Who needs to know which data set?

• What is the cost of collecting, analysing, and presenting data?

• What us the impact of negative or positive data on the enterprise, and the resource/supplier?

• What should be done in the light of the new data?

• Who will take the actions implied/required?

• Who needs to agree that the data strategy is correct? Buy in v apathy/antipathy

Challenges / Considerations / Points to Discuss?• What is a quality level? Who sets / defines it?

• All supply chain to work towards the same goal

• Linking QE results from LSP for different levels with End User Feedback

• External factors affecting the results – e.g. translation quality directly linked to # sales?

• Not all languages can be treated equally – e.g. acceptance of MT in Japan vs Spain

• High quality / perfect really always required?

• Variable quality levels – impact on brand?

• How do high QA scores correlate to user engagement?

• How to customise your metrics based on the communication objectives

• Professional translators offering a ‘less than perfect’ service