Words and numbers: Linguistic analyses of big survey data

Tony McEnery

ESRC Centre for Corpus Approaches to Social Sciencea.mcenery@Lancaster.ac.uk | @TonyMcEnery

Our Data

• NHS Choices• Three areas focused on (comments/responses):

GP practices (14,093,437/5,596,738)Hospitals (8,605,580/4,218,888)Dentists (4,306,698/1,460,343)

• In total we have: 28,971,142/11,692,555• Too much data to be read – reading for comprehension

(200–400 wpm). Over a year (428 days or so) just to read them if you read 200 words a minute for 8 hours a day with no weekends or holiday breaks!

• So we analyse it using corpus linguistics

You are using what?

• Corpus linguistics - using computers to allow us to study large volumes of language – sometimes millions, sometimes billions of words.

• You may not have heard of the approach – but you have almost certainly benefitted from it.

• Very much a British/European innovation which spread to be used more widely.

• We have been pioneering work in this area for over 40 years.

Starting to use a corpus

• You can search for, retrieve and undertake some processing on words effectively.

• Concordance programs allow you to do this swiftly and accurately, presenting data in a way that allows it to be gisted rapidly

• You can then carry out procedures on the data which allow you to gain deeper insight into the use of language itself

• Consider the word ‘cause’

Concordance

Collocation

Keywords

Methodology

Quantitative analysis is combined with

Qualitative analysis

• Corpus linguistics’ big advantage is frequency analysis • but for going deeper into meaning, setting, discourse, no automatic

methods are available• the human mind is needed to work on the data, but the machine

can help!

A typical process:QUANTITATIVE QUALITATIVE QUANTITATIVE

(* Especially in concordance analysis)

Breakdown of the corpus

Word count of patient

comments

Word count of NHS

responses

GPs 14,093,437 5,596,738

Hospitals 8,605,580 4,218,888

Dentists 4,306,698 1,460,343

Pharmacies 690,629 117,858

Care providers 422,133 25,032

Clinics 400,813 110,485

Opticians 179,493 41,994

Acute Trusts 159,385 63,933

Mental Health Trusts 111,557 57,284

Care Organisations 1164 0

Clinical Commissioning

Groups

253 0

Total 28,971,142 11,692,555

Some Results - keywords

an, are, call, do, get, medical, phone, prescription, see, time, when, you

dental, dentist,

ease, emergency,

explained, feel

filling, had, happy

nervous, NHS,

professional, teeth, tooth

visit, years

A&E, admitted,

after, am, caring, clinic,

consultant, had, hospital,

hours, operation, received,

surgery, team, treated,

unit, ward, were

I, !, am, appointment, care, excellent, friendly, have, helpful, me, my, patient, staff, they, told, very

always, appointments, been, good, practice, reception, receptionist, recommend, rude,

service, surgery, this

doctor, doctors, GP, not, nurse, nurses, patients, to, wait, waiting

and, experience, pain, thank, treatment, was

GPs

Hospitals Dentists

Word Frequency across all

comments

Key in which data

sets

good 59,237 GPs and Dentists

excellent 49,090 All

helpful 43,915 All

friendly 42,378 All

rude 29,335 GPs and Dentists

professional 28,104 Dentists

caring 23,719 Hospitals

happy 22,658 Dentists

We should not assume that the positive words are always used positively. For example, the phrase “not caring” occurs 128 times, while “not happy” appears 2309 times.

told, not n’t, get, to

appointmentimpossible, weeks, no, ?

rude, worst, !poor, they, do said, ", you unhelpful, then terrible, asked avoid, awful appalling, even another, was call

be phone

system, on it however

appointments frustrating that, not ?

seem seems

but

book an if or upsee

seem week

receptionist mixed is shame online hit let depends there please OK telephone

but some generally difficult

however good

sometimes overall with improved found the reception bit usually nice happy although improvement improvements pleased

helpful very have

always friendly

great staff

polite

excellent and best practice caring care been professional all highly family service efficient surgery nurses fantastic with recommend team thank years

1 2 3 4 5

Keywords for quantitative

ratings

Word Collocates with a similar meaning Collocates that act

as modifiers

Collocates which are the

target of the evaluation

good None very, really, enough,

overall, not,

generally, pretty,

service, humour(ed), food,

advice, news, doctors,

listener(s)

rude unhelpful, abrupt, arrogant,

unprofessional, dismissive, patronising,

ignorant, uncaring, disrespectful,

obnoxious, condescending, aggressive,

unfriendly, incompetent, impatient,

unsympathetic, impolite, inconsiderate,

blunt, insensitive, obstructive, sarcastic,

unpleasant

very, extremely,

incredibly,

downright, quite,

unbelievably, so,

plain, often,

bordering

staff, receptionist(s),

helpful friendly, polite, pleasant, courteous,

caring, professional, efficient, kind,

understanding, knowledgeable,

supportive, informative, cheerful,

considerate, nice, respectful,

welcoming, accommodating, reassuring,

approachable, sympathetic, lovely,

attentive, obliging

very, always,

extremely, really,

most, so, more,

incredibly, unfailingly

staff, receptionist(s),

doctors, pharmacist

Customer Service• Receptionists are particularly singled out as being rude (although it should be noted

that they are also often described as helpful and friendly)

“Bad mannered and rude Very difficult to get an appointment, very rude staff especially at the reception!”

“…their support staff at the reception desk are awful. They're so rude and seem to think it's the norm to talk down to patients. I've just seen one almost shouting at an elderly patient because she'd got the dates of her appointment wrong!”

“unfortunately the Reception staff at both locations are rude, ignorant and unhelpful - on many occasions I have witnessed patients standing around being ignored whilst the reception staff are laughing amongst themselves.”

• This initial analysis indicates that customer service and politeness tends to be a highly salient aspect of patient feedback.

Collocates - negative Collocates - Positive

waiting room packed, full, crowded,

small, busy, hot,

cramped, stuffy,

overcrowded, tiny, dirty

clean, comfortable,

pleasant, airy, spacious,

quiet

waiting time(s) long, ridiculous,

unacceptable, excessive,

horrendous, appalling,

terrible, poor, lengthy,

joke, atrocious

reasonable, acceptable,

WaitingA potentially interesting keyword is waiting (which is key for both GPs and Hospitals). This keyword tends to occur within two main constructions – waiting room and waiting time(s).

Category Collocates

Body part hernia, cataract, hip, knee, shoulder, eye, bunion,

spinal

Type major, minor, bypass, repair, small, remove

Scheduling cancelled, after, before, following, date, prior,

scheduled, pre, during, first, second, planned

Processes anaesthetic, aftercare

Outcome successful, success

Person surgeon

Operation

• I was very upset that she cancelled the operation and not postponed.

• Administratioin diabolical For the second time in less than 18 months I have had an operation cancelled at very short notice due to disgustingly poor administration.

• However, the other keywords tend to be used in positive contexts, or are merely descriptive:

• The surgeon who performed the operation was very slick and professional and seemed to take no time to cleanly remove the polyp

• After my operation, the aftercare I received from staff at the hospital was fantastic.

• Operation under general anaesthetic was performed after 2 days and I went home the same day.

Keyword Frequency Most frequent 10 evaluative collocates

staff 159,892 friendly, helpful, rude, excellent, lovely, polite, great,

professional, good, fantastic

doctor 106,777 good, excellent, great, rude, helpful, lovely, fantastic,

caring, friendly, brilliant

dentist 73,424 excellent, good, great, friendly, lovely, professional,

fantastic, brilliant, helpful, rude

nurse 41,388 lovely, friendly, excellent, rude, helpful, good,

fantastic, great, nice

receptionist 36, 491 rude, helpful, friendly, lovely, polite, unhelpful, nice,

pleasant, good, excellent

GP 67,483 excellent, good, great, helpful, fantastic, friendly,

happy, rude, caring, brilliant

consultant 14,444 excellent, fantastic, brilliant, wonderful, professional,

great, lovely, friendly, rude, good

StaffTable 6 looks at collocates of the keywords which refer to staff. I have examined the most frequent evaluative collocates for each keyword, and put the negative ones in bold.

Who is rude?

Number

of times

called

rude

%

receptionist 3,056 8.37

staff 6,166 3.85

nurse 402 0.97

doctor 853 0.79

dentist 533 0.72

consultant 65 0.45

GP 303 0.44

Positive and negative words

• What are the key drivers of positive and negative feedback (or what do the words below collocate with?)

Positive Negative

good 59,237 bad 16,945

excellent 49,090 poor 15,274

great 34,298 worst 7,627

best 25,556 worse 7,289

fantastic 15,186 terrible 6,799

brilliant 11,546 awful 6,106

wonderful 10,371 appalling 4,410

amazing 9,749 disgusting 3,246

outstanding 5,019 ridiculous 3,206

exceptional 3,387 useless 2,461

Themes of positive and negative evaluation

• We grouped the most frequent collocates of the positive and negative evaluative words into themes:

THEME Words relating to the theme

TREATMENT care, treatment, dental

SYSTEM system, appointment, time

COMMUNICATION communication

INTERPERSONAL attitude

Proportion of positive to negative evaluation

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Treatment Communication Interpersonal System / Organisation

Negative

Positive

When treatment is evaluated positively

Interpersonal (42%)

Appointments (12%)

Technical competence

(12%)

Communication (10%)

Efficiency (7%)

Cleanliness (6%)

Hard-working (5%)Other (6%)

When interpersonal skills are evaluated negatively

Impolite / rude (41%)

Apathetic (9%)

Lazy (7%)

Not listening (7%)

Abrupt (7%)

Don't answer phone (6%)

Appeared impositioned (5%)

Not smiling (5%) Other (11%)

Proportion of positive to negative evaluation

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Dentists Paramedics Midwives Nurses Opticians Pharmacists Doctors Receptionists

General Positive and Negative Descriptors

Positive Negative

Some negatively evaluated receptionists

• The last and final time i tried to make an appointment at this Health Centre, the receptionist laughed in my face

• After speaking to a very snooty receptionist who was very unhelpfulwe are no further to getting the splints removed.

• Receptionist interrupted talked over me on the phone, had to ask if I could finish my sentence three times.

• Upon approaching the desk I had to wait whilst the receptionist finished telling their colleague about their weekend, then when I asked my question I was given a response of "well it 's not exactly hard to work out ".

• Receptionist had no compassion whatsoever.

• My daughter is now too embarrassed to go back when she was asked personal questions about the nature of her ailment by the receptionist .

Why do receptionists do so badly?

• Encountered by largest number of people?

• Gatekeeping role has capacity to annoy – the “face” of systems they didn’t design

• Questions mistaken as nosiness

• Is social class a factor? E.g. viewed as relatively inexpert so afforded less respect

• Is gender a factor? (94% of British receptionists are female)

Keyword Frequency Most frequent 10 problematic verbs (where available)

staff 159,892 refused, failed, ignored, complained, lost, shouted, forgot,

insisted, laughed, cancelled

doctor 106,777 refused, failed, insisted, complained, ignored, laughed,

shouted, dismissed, forgot, shrugged

dentist 73,424 refused, failed, insisted, missed, cancelled, ignored, shouted,

complained, laughed, hit

nurse 41,388 refused, shouted, insisted, complained, admitted, forgot,

failed, ignored, laughed, threw

receptionist 36, 491 refused, insisted, shouted, ignored, hung up, complained,

laughed, shrugged, failed, barked

GP 67,483 refused, failed, ignored, insisted, forgot, complained, missed,

dismissed, laughed, shouted

consultant 14,444 refused, dismissed, ignored, failed, admitted, complained

What about problematic behaviour? Below are the 10 most frequently mentioned past tense verbs which are evaluated as negative behaviours for each of the staff keywords.

• The receptionist barked out questions and orders with barely looking at me

• One Doctor laughed at me when I went in crying with Acne at age 25 ...

• As soon as she woke up she had to vomit and the nurse threw a vomit bowl at her and did not even bother to help her .

• I asked what I should do if the Dr doesn't arrive and the receptionist shrugged their shoulders and stated they didn't know .

Conclusion

• Our techniques allow large bodies of textual data to be exhaustively analysed

• The exhaustive analysis can show trends clearly

• Actors, actions, and attitudes shine through

• Some results will confirm and reinforce views, others will undoubtedly surprise

• Early days – much more work to be done. Other data sets to look at as well in the future.