Upload
vanjul-christian-hutajulu
View
222
Download
0
Embed Size (px)
Citation preview
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
1/36
1
NEWS FEED RATER GUIDELINES
Introduction 2
An Overview of the Rating Process 3
Things to do BEFORE rating each story ......................................................................................................... 3
Things to do WHILE rating each story ........................................................................................................... 3
Rating Explanations 5
Overall Rating: How much do you want to see this in your News Feed? ..................................................... 5
3 Considerations for Overall Rating: Person / Page, Content, and Impact ................................................... 6
Person / Page and Content ........................................................................................................................... 7
Person / Page ............................................................................................................................................ 7
Content ...................................................................................................................................................... 7
Impact ......................................................................................................................................................... 12
Type of Impact ........................................................................................................................................ 12
Amount of Impact ................................................................................................................................... 13
How to use Person / Page, Content, and Impact together ......................................................................... 15 Extra factors related to story ratings .......................................................................................................... 16
Repetition ................................................................................................................................................ 16
Reading Articles, Watching Videos, and Viewing Photo Albums ............................................................ 16
Overview of Ratings .................................................................................................................................... 17
Instructions for Comments
–
First Section 18
Describing Person/Page .............................................................................................................................. 18
Describing Content...................................................................................................................................... 19
Describing Impact (Type and Amount) ....................................................................................................... 19
Describing 2’s and 1’s.................................................................................................................................. 20
Additional Questions 21
Content type: Friend/Family, Public, or Both? ........................................................................................... 21
How much does this story make you feel entertained? ............................................................................. 24
How much does this story help you feel connected to family/friends? ..................................................... 25
How much does this story help you feel informed about the world? ........................................................ 26
Instructions for Comments – Second Section 28
Submitting Your Responses: “Submit”, “Skip”, and “Submit (no share)” 30
When to “Submit” ....................................................................................................................................... 30
When to “Skip” ........................................................................................................................................... 30
When to “Submit (no share)” ..................................................................................................................... 30
Instructions for “Reorder” and “ akeoff” Tasks
31
Where and When to access these tasks ..................................................................................................... 31
Reorder ....................................................................................................................................................... 31
Bakeoff ........................................................................................................................................................ 31
Printable Reference Materials 33
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
2/36
2
Introduction
Overall Project Goals
Understand how people feel about the stories they see in their News Feed. Understand which stories people want to see in their News Feed, and why.
Why are we using this rating system?
By using this rating system, we’ll be able to understand which stories you think are the very best and
which you think are the very worst. Second, we’ll understand the difference between just good enough
to be in feed and just bad enough to not be there.
An Honest Approach
Your feedback should be as honest as possible. There’s no judgment about what you like to see and
don’t like to see in your News Feed! The more honest the feedback, the more helpful it is.
About this Document
While this document is long, it contains the information necessary for you to complete your ratings.
There are many different types of stories you may see on Facebook, and we’ve created guidelines to
help you give feedback about the stories you see in a thoughtful, systematic way.
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
3/36
3
An Overview of the Rating Process
Things to do BEFORE rating each story
View the story as you normally would if you saw it in your News Feed to get a naturalistic first
reaction.
Like, comment, share, click, or do anything with the post that you normally would do in everyday
life.
o
It is very important to do this BEFORE giving the post a rating.
o
DO NOT do any of these actions unless you would normally do so on your own time.
o
You can interact with the post from the rating tool.
Things to do WHILE rating each story
Use this rating guide and Printable Reference Materials to determine what numerical ratings to
give a story.
Provide open-ended feedback in the comment boxes to help explain why you rated a story the
way you did.
Complete the additional questions for each story and be sure to reference this rating guide for
these questions as well.
STEP 1
Interact as you
normally would
STEP 2
Complete your
ratings
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
4/36
4
What to do AFTER you are done with rating all stories for the day
After you have finished your 1 batch of 60 stories, complete the “Bakeoff” and “Reorder” tasks,
explained later on in this document.
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
5/36
5
Rating Explanations
Overall Rating: How much do you want to see this in your News Feed?
Definitely do
NOT want to see
in Feed
Do not want to
see in Feed
Don’t mind
seeing in Feed
but wouldn’t
mind missing
either
Want to see in
Feed
Definitely want
to see in Feed
1 2 3 4 5
The main factor in considering how you rate a story is ultimately whether or not you want to see this
story in your News Feed. For some stories, it may be very clear whether you want to see them in your
Feed (“5”) or do NOT want to see them in your Feed (“1”). For other stories, you may feel less strongly
about them but still want to see them (“4”) or NOT see them (“2”). And, for some stories, you may feel
ambivalent about whether or not you would want to see them (“3”). In other words, a “3” means it
would be okay if the story were in your Feed, but it would also be okay if the story were not in your
Feed.
5’s should be reserved for the best stories, and 1’s should be reserved for the worst stories. This will
help us to best differentiate between stories that are good vs. stories that are the best (and stories that
are bad vs. stories that are the worst). Therefore, it is likely that you will be choosing 5’s and 1’s less
often than you will be choosing 2’s, 3’s and 4’s.
When rating a post, in addition to considering the main part of the post, you may also take into account
additional information like how many likes and comments a post has, who has liked or commented, orthe nature of the comments. In other words, consider everything you see on the screen in the rating
tool as you normally would consider these aspects of a post in everyday life. If you don’t look at these
things normally, you don’t have to look at them when rating.
Also, we realize that you may have seen some of these posts before in everyday life (e.g., on Facebook
but not while doing your work for Appen) and some you may be seeing for the first time when you rate
them. For posts you’ve seen before, please try to recall your feelings and reactions from the first time
that you saw this and channel this into your ratings. For posts you have not seen before, please be sure
to take the time to make your first impression before you go through the rating process (“Things to do
BEFORE rating each story”).
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
6/36
6
3 Considerations for Overall Rating: Person / Page, Content, and Impact
In order to arrive at your final score, and to help you mentally work toward your overall rating, you
should take 3 aspects of the story into account:
Person / Page: How much do you care about this person or Page?
Content: How interested are you in seeing this specific content?
Impact: How much of an impact does seeing this have on you?
You must choose a numerical rating for each of these 3 aspects of a story, which will then feed into your
overall rating.
Details about how to approach these questions are discussed in the next section.
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
7/36
7
Person / Page and Content
Person / Page
Someone youreally don’t care
about
Someone youdon’t care about
Someone yousomewhat care
about
Someone youcare about
Someone youreally care about
1 2 3 4 5
Content
Content you are
really notinterested in
seeing
Content you are
not interested in
seeing
Content you are
somewhatinterested in
seeing
Content you are
interested in
seeing
Content you are
really interested
in seeing
1 2 3 4 5
For Person / Page and Content, there is no right or wrong way to feel. Considering these elements
separately allows you to give high or low overall ratings based on how much you care about the person
(or Page) posting OR based on how much you are interested in seeing the content. There may be posts
from people you care about that contain content you don’t care about. Or, there may be posts from
people you don’t care about but that contain content you do care about. However, there may be manytimes for which your feelings about the person / Page and content match up. That’s fine too!
For Page, you can think about “how much you care about the Page” in a couple different ways. First of
all, Pages can be businesses or brands, news organizations, celebrities, public figures, or any other non-
friend on Facebook. So, you may care about a Page more if you tend to like most of the Page’s content,
or you may like it less if you only like some of the Page’s content. You might also consider how much you
like the Page, how much you trust the Page, or anything else that helps you make an assessment about
how much you care about the Page.
In addition, sometimes there may be multiple people or a person and a Page featured in one story. Here
are examples of some of these, and what you should do in each case:
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
8/36
8
The person posting is someone you don’t know.
o If you see a story where a friend is tagged or featured in a post by someone else on
Facebook that you don’t know, please give the Person rating based on the person you
do know in the post.
o
Example:
If you know Dan (who
was tagged in a photo)
but not Mike, your
Person rating should be
about Dan.
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
9/36
9
You know both the person who posted as well as the person who was tagged in a post.
o If you see a story where a friend is tagged or featured in a post by another one of your
friends, your “person” rating should be about the friend who is tagged or featured in the
story .
o
The fact that a story might feature an interaction between two of your friends should be
considered in the “content” rating.
o Example:
If you know Ben (who
was tagged in a photo)AND Steve, your Person
rating should be about
Ben, who was tagged.
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
10/36
10
You see a post that displays multiple stories at the same time.
o If you see an “aggregate” post (a post where there are two or more people / Pages that
have done something, you should provide your Person rating based on the person in the
story who has the most impact on you.
o
Example: If two of your friends shared an article and you see ONE story about those two
shares, you would probably rate Person based on the friend who is more meaningful to
you in this context.
o Example: If you see a story about multiple people posting to your friend’s timeline for
his/her birthday, you should rate Person based on the person who is receiving those
Choose between
Cameron and Lindsay
for the Person rating by
choosing the friend who
is more meaningful to
you in this context.
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
11/36
11
posts (the person whose birthday it is), since that is the main reason why you are seeing
that “aggregate story.” (see example below).
o For the Content rating, you should take into account the fact that you are seeing
multiple stories in one and report on how you feel about that content as a whole.
Additional scenarios
There are two or more people you know tagged in a post.
o
For this type of story, you should rate the person who is more meaningful to you in this
context.
Someone posted something in a group.
o For this type of story, you should rate the person who posted in the group, not the
group itself. If this post was interesting to you or had impact on you because it was in a
particular group, you should take that into account in your Content and Impact ratings
and discuss the role of the group in your comments.
The Person rating
for this story
should be based
on Annie and not
based on the
people posting on
her timeline.
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
12/36
12
Impact
In addition to Person and Content, you must consider: How much impact does this story have on you?
Impact must be considered for every story, and should be derived directly from the Person / Page
and/or Content rating.
For impact, you will provide 2 pieces of feedback: Type of Impact and Amount of Impact.
Type of Impact
You use this information in discussions with people, online or offline.
Examples include:
Talking about what you saw or read with the person who posted.
Talking about what you saw or read with someone else you know.
Sharing the post with others, either through Facebook or via another method.
You gain information that you want to have.
Examples include:
Seeing information about a friend or family member, which can help you stay connected.
Learning about a news story, product, person, event, or something else that is interesting,
informative, useful, valuable, or relevant to you.
Bookmarking the information or saving it in some way.
You have a (desired) emotional reaction:
Examples include:
Feeling delighted, joyful, grateful, proud, excited, or happy.
Laughing or being highly entertained.
Feeling empathy for someone who had a bad day.
Emotions can be negative as well, as long as it is an emotion that you want to feel. For example,
you could feel sad about a friend’s parent passing away but be glad that you saw that sad story,
or feel anger at information in a news article but be glad that you know what happened.
If you feel bad or sad or upset but you still want to see the post in your News Feed because you
are ok with feeling this way, then you should report that you had a desired emotional reaction.
In other words, this type of impact captures the fact that you had an emotional reaction that
was wanted or acceptable in some way . If you feel bad or sad or upset and do not want to feel
this way, then you should not choose “desired emotional reaction” as impact type and choose a
“1” for Amount of Impact, which in this case would be negative. (See next section for more
details on Amount of Impact.)
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
13/36
13
N/A
You should choose “N/A” when there is no impact (2) or a negative impact (1) on you. For instance, if
you have an undesired emotional reaction, you should choose “N/A” for impact type. If you have an
undesired emotional reaction, you should discuss this in your comments.
If you select “N/A” for Type of Impact, it does not make sense to then choose “Has a little impact onyou” (3), “Has an impact on you” (4), or “Has a significant impact on you” (5) for Amount of Impact.
Remember that if there is any impact, you must identify the type. If there is no impact, then Type of
Impact should be “N/A.”
Amount of Impact
Really does not
have an impact
on you / has a
negative impact
Does not have
an impact onyou
Has a little
impact on you
Has an impact
on you
Has a significant
impact on you
1 2 3 4 5
The Amount of Impact score directly relates to the Type of Impact score in that it reflects how much of
an impact a story had on you.
For instance if you had a desired emotional reaction but it was small or at a low level, the
amount of impact would be 3 (e.g., a chuckle). If you had a desired emotional reaction that was
stronger, amount of impact would be 4 (e.g., a laugh). If you had a really big emotional reaction,
amount of impact would be 5 (e.g., laughed hard for a few minutes).
Or, imagine you picked “You use this information in discussions with people, online or offline.” If
you thought you might use the information in discussions, amount of impact would be 3. If you
thought you probably would use the information in discussions, amount of impact would be 4. If
you were very certain you would use the information in discussions, amount of impact would be
5.
Or, imagine you picked “You gain information that you want to have.” If you thought the
information was a little useful/helpful/etc., amount of impact would be 3. If you thought the
information was useful/helpful/etc., amount of impact would be 4. If you thought theinformation was very useful/helpful/etc., amount of impact would be 5.
Stories should be rated a “5” if they had a significant impact on you in one of the ways described in the
Type of Impact section. The less impact on you, the lower the impact score.
What does a “1” impact mean? (“Really does not have an impact on you / has a negative impact”). A
negative impact includes anything that detracts from your experience on Facebook or makes your
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
14/36
14
experience worse. If you use words like “annoying” or “offensive”, it should be a 1 on impact. So, the
difference between a 2 and a 1 on impact is that a 2 means “no impact” but a 1 is either a more extreme
version of no impact, or impact that is negative.
Note: only one form of impact needs to be met in order for there to be impact. Not all of these criteria
must be met, just impact in one of the ways mentioned above.
Multiple Types of Impact
In some cases, a story may be associated with multiple types of impact. The Type of Impact question in
the rating tool allows you to choose multiple types of impact:
You may choose multiple types of impact if they apply, but please think carefully about whether morethan one type of impact applies, and refer to your rater guide for the distinctions between impact types.
In addition, if there are multiple types of impact you should rate Amount of Impact based on the highest
level of impact that exists. Here are some examples:
Example:
If “gain information” is a 3 on Amount of Impact and “emotional reaction” is a 4 on Amount of
Impact, the Amount of Impact score = 4.
Example:
If “gain information” is a 5 on Amount of Impact and “emotional reaction” is a 3 on Amount ofImpact, the Amount of Impact score = 5.
Example:
If “gain information” is a 4 on Amount of Impact and “emotional reaction” is a 4 on Amount of
Impact, the Amount of Impact score = 4.
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
15/36
15
Note: You should NOT average or add the Amount of Impact scores together when there are multiple
Types of Impact. Simply choose your Amount of Impact score based on the highest amount of impact
that exists.
How to use Person / Page, Content, and Impact together
The flow for the ratings should be:
1)
Consider Person / Page and Content.
o In many cases, these ratings will be aligned but in some cases they might differ.
2)
Consider Impact.
o You should be able to draw a clear line between caring about the person / Page
and/or being interested in the content and how much impact it has on you.
o
Depending on the level of impact, however, your Overall rating may go up or down.
o Remember, you will need to record both the type (3 choices or N/A) and amount of
impact (1 through 5 score).
3)
Consider the Person / Page, Content, and Impact scores in relation to the Overall score.
o Verify that these scores align with how much you want to see the story (Overall score).
o If they don’t align, reconsider Impact, which should have the most direct relationship to
the Overall score. (In other words, Person / Page or Content should translate directly to
Impact, which should translate directly to how much you want to see a story).
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
16/36
16
Extra factors related to story ratings
Repetition
We know that some people post more often than others, and that seeing multiple posts from a person
or Page recently may make you less likely to want to see similar content from that person or Page in the
future. You may take this into consideration. However, if multiple different sources post, for example, asimilar article, please DO NOT consider this when rating for each day.
For example, imagine that multiple friends or Pages are posting articles about a recent news event (e.g.,
Malaysia Airlines missing plane). Please rate each of these stories as if you had not seen other similar
stories.
When you begin a new day of ratings, you are allowed to consider repetition of stories that you saw
from the previous day. In other words, if you see multiple stories on a specific topic that you also saw
stories on the day before, you may consider the repetitiveness of the content when completing your
ratings. If repetition plays a role in your assessment of the story, please describe this in your comments.
Reading Articles, Watching Videos, and Viewing Photo Albums
Articles
When you see a post with an article in it that you would typically read , please go ahead and read
or skim the article (for up to 5 minutes) before making your final rating.
You should take into account how much you are interested in the actual content, and how much
impact it has on you, before you make your final rating about whether or not you want to see it
in your News Feed.
Videos
The same approach (from reading articles) applies to watching videos.
When you see a post with a video in it that you would typically watch, please go ahead and
watch the video (for up to 5 minutes).
Photo Albums
The same approach (from reading articles) applies to photo albums.
If you see a post with a photo album in it that you would typically view, please go ahead and
view some of the photos in the album before making your final rating.
Note: While it is important that you look at the articles, videos, and photos you’re interested in before
completing your ratings, remember that you also must complete your ratings within the time
provided. Please take this into account as you look at content throughout the day.
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
17/36
17
Overview of Ratings
Score 1 2 3 4 5
Person / Page
Someone you
really don’t careabout
Someone you
don’t care about
Someone you
somewhat careabout
Someone you care
about
Someone you
really care about
Content Content you are
really not
interested to see
Content you are
not interested in
seeing
Content you are
somewhat
interested in
seeing
Content that you
are interested in
seeing
Content you are
really interested in
seeing
Impact
Really does not
have an impact on
you / has a
negative impact
Does not have an
impact on you
Has a little impact
on you
Has an impact on
you
Has a significant
impact on you
Overall Rating Definitely do NOT
want to see in
Feed.
Do not want to see
in Feed.
Don’t mind seeing
in Feed but
wouldn’t mind
missing either.
Want to see in
Feed.
Definitely want to
see in Feed.
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
18/36
18
Instructions for Comments – First Section
In addition to providing a numerical rating, please indicate why you chose the rating you did. Please
include details that help to further describe the choices you made about rating the Person / Page,Content, and Impact. For instance, you might give feedback about why the person is important (or
unimportant) to you, or why the topic was something you really did (or did not) want to know about.
You could have many other reasons for why you would or would not want to see a story in your News
Feed. Please use the open-ended Reason box to provide additional insights as to why you chose the
rating you did.
We recommend that you describe your feedback in your own words, but we ask that you also
supplement your explanation with words from the guidelines to make clear how you’ve decided on your
ratings. Here’s an example:
“This is a post from a high school friend of mine that I still keep in touch with and care about. She
posted an article about knitting and I like knitting. After looking at the article I learned
something new about knitting, which was cool. So the content was interesting to me and I
gained information that I wanted to have. I want to see this story in my Feed.”
Your explanations should correspond to your numerical ratings. Also, please be sure to discuss Person /
Page, Content, and Impact in your open-ended explanations. All three must be addressed. This will both
help your team leads to best understand your thought processes, as well as assist you in thoroughly
capturing how you feel about a story.
Add your personal touch to each comment
Then tie back into the guidelines
Others should be able to determine what your numerical scores were based on the quality ofyour comments
You must explain why you chose the rating you chose for Person / Page, Content, and Impact. Using
words like “because” and “since” in your sentences is a helpful way to ensure you explain why you feel
how you feel.
Describing Person/Page
Here are some ways you might describe why you care or don’t care about the person or Page you are
rating:
“ I d on’t know anything about this page and never visit it so this is a page I really do not care
about .” (Person = 1)
“ This person is someone I don't care about because she's an acquaintance from high school that I
don't speak to.” (Person = 2)
“ This is from a local friendly acquaintance who I somewhat care about.” (Person = 3)
“I care about this Page because it’s a TV show I like.” (Person = 4)
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
19/36
19
“This is someone I really care for since she is my sister who I am very close with.” (Person = 5)
Describing Content
Here are some ways you might describe why you are or are not interested in the specific piece ofcontent you are rating:
“T his article is about this diet that is actually really unhealthy which is something that I am really
not interested in seeing in my feed because I really hate it – it’s obnoxious.” (Content = 1)
“ The content is not that interesting to me because it ’s a recipe with olives in it and I don’t like
olives or really cook that much.” (Content = 2)
“ This content is somewhat interesting to me since I like this basketball team a little and I like to
see posts about the team.” (Content = 3)
“ This story features the life of a musician which is interesting to me because I used to be a
musician as well.” (Content = 4)
“ This is a photo of my young niece who lives far away. I am really interested in seeing thisbecause I don’t get to see her very often and I love to see her sweet face and that she is doing
well!” (Content = 5)
Describing Impact (Type and Amount)
Here are some ways you might describe why the story you are rating does or does not have an impact
on you:
“This political joke had a negative impact on me since I was offended by the joke and the
language it used.” (Type of Impact = N/A, Amount of Impact = 1) “This has no impact on me because I didn’t take this quiz and don’t plan to share it with others.”
(Type of Impact = N/A, Amount of Impact = 2)
“This post had a little impact on me since it made me feel slightly happy to see that her dog was
feeling better.” (Type of Impact = have a desired emotional reaction, Amount of Impact = 3)
“This post had an impact because I learned a good amount of information about this country.”
(Type of Impact = gain the information that you want to have, Amount of Impact = 4)
“This had significant impact on me because I’m definitely going to talk about these tips for
communicating tonight with my wife. (Type of Impact = use this information in discussions with
people, Amount of Impact = 5)
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
20/36
20
Describing 2’s and 1’s
In some cases, there will be people or Pages you don’t/really don’t care about, content that you are
not/really not interested in seeing, and posts that don’t/really don’t have an impact on you. These will
be rated with 2’s or 1’s for the corresponding questions. Though it may be difficult, you should still
explain why you don’t care about a person or Page, why you’re not interested in a particular piece of
content, or why something does not have an impact on you. Here are some examples:
Person/ Page: “Someone you don’t care about”
“This is an old coworker that I don’t care about because I never see them.”
“This is someone I met once at a party and I don’t care about them because they are not an
actual friend.”
“This is someone I went to school with and was never really close with - I don’ t care about him
because it doesn’t matter to me what happens in this person’s life.”
“I don’t care about this Page because it’s for a product I used to like but don’t like anymore.”
“I don’t care about this Page because it’s a celebrity that doesn’t interest me.”
Content: “ Content you are not interested in seeing”
“I don’t find this post interesting because I don’t care to see the art that this friend likes.”
“This post isn’t interesting to me because this story is about bowling and I think bowling is
bor ing.”
“This doesn’t interest me because it’s a blog post about a show I don’t watch and am not
interested in watching.”
Impact: “Does not have an impact on you”
“ This had no impact on me since I probably won't take part in the conversation.”
“ Since I am not planning on doing anything with this information, this info. is not useful to meand it does not have an impact on me.”
“It doesn’t impact me because I will not be playing this video game in the future and have no
emotional reaction to seeing this.”
** IMPORTANT **
Be sure to enter this information in the Comment box on the left hand side of your screen, not in the
comment box of the actual News Feed story, in the middle of your screen. Otherwise, your comments
describing your ratings will be shared on Facebook (yikes!)
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
21/36
21
Additional Questions
In addition to the main rating, we’d like you to answer some additional questions about each story that
you see.
** IMPORTANT **
This additional set of questions is NOT related to the previous set of questions. You should consider
these questions to be separate and should not worry about aligning your answers to these additional
questions with your answers to the first set of questions.
Content type: Friend/Family, Public, or Both?
For this question, we’d like you to categorize which type of content the story contains – content about a
friend or family member, “Public” content, or both? In general, content that is Public is anything posted
by a Page or other entity like a news organization or celebrity. On the other hand, Friend / Family
content is produced by people you know.
In general, if a friend shares something that is public content, you should rate this as “Public.” However,
if a friend shares something that is Public content but he/she also adds additional personal information
in the post that makes the overall story seem personal, you should rate that as “Both.” In other words,
you should make a distinction between articles (or other pieces of Public content) where the main
information you gain is from the article vs. articles where you both learn something from the article AND
about your friend who shared it. See below for examples.
Friend/Family Content
Stories about people you know personally
Photos, status updates, or videos created and posted by a friend or family member
Updates from groups you belong to
Public Content
Stories about public things or people you don’t know personally
Content from a Page (e.g., public figure, business, news organization)
News photos, articles, or videos
Celebrity content
Friend shares of “Public” content where the information you gain is about the content and
NOT about your friend
Both
Friend shares of “Public” content where the information you gain is about BOTH the content
AND your friend
Friend shares of “Public” content where:
o You learn more about your friend
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
22/36
22
o You learn how your friend feels about something
o You feel closer to your friend
o For you, the post is more about your friend than the public content that is being
shared
o Here is an example to help explain the when you would pick “Public” vs. when you
would pick “Both”:
For the two stories above, the one on the left is an article shared by a friend with not much additional
context or commentary. It likely does not provide much information about the friend – it says “nice
narrative.” The story on the right, however, might reveal to you that your friend is very passionate about
cycling, or that he/she is planning to vote in an upcoming election. Since there is information about a
friend alongside a post from a public Page that the friend is sharing, this could be considered “Both.”
Probably
“Both”
Probably
“Public”
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
23/36
23
In addition, here’s a decision tree you should use to help you decide whether a story is “Friends /
Family”, “Public”, or “Both.”
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
24/36
24
How much does this story make you feel entertained?
There are many types of stories you might see on Facebook. One aspect that we’d like you to consider
for each story is – how entertained did the story make you feel? You should use the following scale to
answer this:
Really notentertained
Notentertained
A littleentertained
Entertained Reallyentertained
1 2 3 4 5
For this question, we want you to only consider stories that are about entertainment-style content.
Examples of entertainment-style content are:
Movie trailers, music videos, TV show clips
Videos, articles, photos, and other media (e.g., memes, gifs) that make you laugh or amuse you
Things that are lighthearted in nature
Things that are jokes, are silly, or “just for fun”
Things that are NOT necessarily entertaining are:
A nice picture of your friend
An interesting article
A useful update about something, like an event
However, just because a story features entertainment-style content doesn’t mean that you feelentertained by the story. So, if you see something like a meme or a TV show clip that doesn’t make you
feel entertained, you should choose “Not entertained” (2).
Stories should be rated “Really not entertained” (1) if something gives you a negative feeling. For
instance, if a post was intended to be humorous but you felt it was in poor taste or offensive, you should
rate it “Really not entertained.”
You may find that you’ve already considered how entertained you were by a story when talking about
type of impact (desired emotional reaction). That’s ok! This is a more specific way for us to understand
which stories make you feel entertained.
** IMPORTANT **
“Entertaining” should be considered as a different dimension from “interesting.”
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
25/36
25
How much does this story help you feel connected to family/friends?
One of the things that people use Facebook for is to stay connected with friends and family. In addition
to telling us how much you care about the person related to the story, we’d like you to consider for each
story – how connected to family/friends did this story make you feel? You should use the following scale
to answer this:
Really not
connected
Not
connected
A little
connectedConnected
Really
connected
1 2 3 4 5
By feeling connected to family and friends, we mean a feeling more than just“I'm reminded that this
person is around” or “I like seeing these people on Facebook.”
You don’t have to actually connect to / communicate with someone for this question to get a high rating
– it’s more about how connected you feel to the person by seeing the information inthe story.However, if you do communicate with the person because of seeing the story and that makes you feel
connected, feel free to take that into account.
By feeling connected to family and friends, we mean that you feel closer to the person or people in the
story, whether by learning something about them, interacting with them, or enhancing your relationship
with them in some way.
Stories should be rated “Really not connected” (1) if something gives you a negative feeling. Forinstance, if the story was about family or friends but ended up making you feel alienated, disconnected,
irritated, or offended, you should rate it “Really not connected.”
This question is completely separate from the previous set of questions - don't worry if you give
something a 4 or a 5 for your Overall rating but do not rate it as a 4 or 5 for this question.
Note: This question is ONLY about feeling connected to family and friends. This question is NOT about
feeling connected to a Page, to a celebrity, to a news story, groups of people you don’t actually know,
or to any other entity. Accordingly, if the story is from a Page, it should most likely be rated “ Not
connected ” ( 2).
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
26/36
26
How much does this story help you feel informed about the world?
Again, there are many types of stories you might see on Facebook, and we’re interested in
understanding the different aspects of each story. Some stories in News Feed can help inform you, teach
you something, or keep you up-to-date about the world around you. In this case, “world” could mean as
close to you as something in your own neighborhood or as far from you as something going on in
another country.
For this question, we want you to consider – how informed about the world does this story make you
feel? You should use the following scale to answer this:
Really not
informed
Not informed A little
informedInformed
Really
informed
1 2 3 4 5
The main thing to remember for this question is: we are NOT asking about information about friends,
family, or people you know. (“ How much does this story help you feel connected to family/friends? ” is
the question that focuses on people you know.) In addition, “informed about the world” does NOT mean
the same thing as “interesting” or “entertaining”.
Here are some examples of things that might make you feel informed about the world:
A local, US, or World news story
A story about weather, sports, or celebrities
A human interest or feature story
An upcoming movie or product release
An event going on in your town
Something instructional (e.g., recipe, “how to”)
A friend discussing any of the above
Here are some examples of things that probably would NOT make you feel informed about the world (in
the way we’re asking about it here):
A post from a friend with a picture of his/her new baby
A post that a family member is not feeling well
A post about a friend’s relationship, birthday, or pet
A picture or video that is funny but doesn’t give you any information
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
27/36
27
As usual, if the story is something like an article or video, only read or watch the content if you would do
so in real life. If you don't feel informed just by seeing the post in your News Feed that's fine. Similarly, if
you don't feel informed even after reading or watching the content, that's fine too! It's all about how
informed you feel.
In addition, if you see something like a news story that provides information but you’re not personally
interested in it and don’t get any information that is valuable to you, then you should choose “Not
informed” (2). In other words, just because a story may provide information does not mean that you will
feel informed by it.
Stories should be rated “Really not informed” (1) if something gives you a negative feeling. For instance,
if the story is misleading, provides false information, makes you feel like you wasted your time, or
presented information that was confusing or irritating to you in some way, you should rate it “Really not
informed.”
A couple other things to keep in mind:
Remember the repetition rules. If you already have the information you see in a story, then it is
okay to say you don’t feel informed by it. However, if you got the information from a story you
rated the same day, please rate the story as if you did not already have the information.
You may have noticed that this question is similar to the type of impact “information you want
to have.” We recognize this. Please try to consider these questions separately. You do not need
to make sure that your impact rating and your “informed” rating match perfectly, just pick the
rating that reflects how much the story makes you feel informed about the world.
** IMPORTANT **
With all of these questions (“entertaining”, “connected”, “informed”), you should not feel compelled
to report that something makes you feel entertained, makes you feel connected to family/friends, or
makes you feel informed about the world if you do not feel that way. It is more important to be
honest in your responses. We expect that these questions will not apply to each story you rate. As
long as you are honest and thoughtful, and explain your choices in the comments (see next section),you are doing the right thing.
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
28/36
28
Instructions for Comments – Second Section
In addition to writing comments for the first set of ratings (Person / Page, Content, Impact, and Overall
Rating), you will provide comments to explain your responses on the 3 additional questions mentioned
previously (“ How much does this story… make you feel entertained, help you feel connected to
family/friends, help you feel informed about the world?” )
As with the first section’s comments, your comments should be in your own voice but should reference
the specific questions you answered. The comments should focus on answering the “why” of the
responses you chose. Here are a few examples:
“This post made me feel really entertained because I watched the video of the little
girl dancing and was amazed by how good she was, plus it was fun to watch. It didn’t
make me feel connected to the person who posted it because it has nothing to do with
her personally, and overall this doesn’t make me feel informed.”
(Entertaining = “Really entertained” , Connected = “Not connected” ,
Informed = “Not informed” )
“This post was more serious so it didn’t make me feel entertained, but it did help me
feel really connected to my cousin. She is moving across the country so me seeing a
picture of that was good because I now know about a big moment in her life. This post
is about my cousin, so it didn’t make me feel informed about the world.”
(Entertaining = “Not entertained”, Connected = “Really connected”,
Informed = “Not informed”)
“This post does nothing for me – I did not feel entertained because I don’t find this
topic amusing , I don’t feel connected to the person who is in this picture since he didn’treveal much about himself, and it didn’t make me feel informed.”
(Entertaining = “Not entertained”, Connected = “Not connected” ,
Informed = “Not informed” )
Similar to the comments in the first section, you need to explain why you’ve rated something a 1 (e.g.,
Really not entertained, Really not connected to family/friends, Really not informed about the world.
Here are some examples:
Entertained = 1 (Really not entertained)
o “This post was a meme that was supposed to be funny but I found it offensive to the
group of people it was about which gave me a bad feeling, so I was really not
entertained .”
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
29/36
29
Connected = 1 (Really not connected)
o “ This post was from my friend trying to look cool but he comes off as not genuine.
This is annoying to me and really does not make me feel connected to him.”
Informed = 1 (Really not informed)
o
“ This story is trying to spread information that is just wrong. I hate that people outthere will see something like this. This is the opposite of making me feel informed – I
really do not feel informed about the world by seeing this.
Please note, it is not okay for you to use the exact language in the examples above. Instead, you should
find your own words to explain why a specific post may not entertain you, help you feel connected to
friends and family, or matter to your life.
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
30/36
30
Submitting Your Responses: “Submit”, “Skip”, and “Submit (no share)”
When to “Submit”
After you have completed all of the ratings, comments, and questions for a story, you can click “Submit”.
Note: you cannot go back to a story after you have submitted it.
When to “Skip”
As part of ongoing checks, your comments may be reviewed for quality from time to time by Appen
team leads. No other raters will ever see your posts or your ratings.
There may be times where you do not feel comfortable providing ratings on a particular story. The
“skip” option should be used only when you are very uncomfortable providing a rating for a story at all.
“Skip” should NOT be used when:
You’re having a hard time deciding
If you find content offensive or bad (like a gruesome photo). In this case, you should go
ahead and rate that a 1.
When the “rate but don’t share” option will work (see next section)
You should also use “Skip” when you see an error. This includes times when a story does not render
properly in the tool or when you encounter some other type of technical error. We will provide
instructions on how you should report problems in a separate document.
Also, you may have noticed that when you post something to Facebook, it often shows up in your own
News Feed. We will try not to show you your own posts in the tool but if you do see a post that you
yourself made, please do not rate it but instead, skip it.
In addition, if you see a post that you think is a repeat, please DO NOT skip it. Sometimes a post may
look like a repeat but it is actually a new story (e.g., a post you’ve previously seen that has a new like or
comment on it). Again, please rate these stories – do not skip them.
When to “Submit (no share)”
There may be some instances, however, where you feel comfortable providing a rating on a post but do
not feel comfortable sharing it with your Appen team leads. In this case, we ask that you provide a rating
so that your feedback can be analyzed anonymously, in aggregate. No one will ever discuss the ratingswith you if you click “submit (no share)”.
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
31/36
31
Instructions for “Reorder” and “Bakeoff” Tasks
At the end of the day, and after a break from completing your last story rating you will do 2 additional
tasks called “Reorder” and “Bakeoff”, which are explained below.
Where and When to access these tasks
On the main rating page, there will be links to the Reorder and Bakeoff tasks.
** IMPORTANT **
You may only click on the Reorder and Bakeoff links at the end of the day, after you have completed
rating all of your stories. You are not allowed to click on these links until the end of your workday.
Reorder
At the end of each day of rating (after rating 1 batch of 60 stories), you will view a group of stories on
one screen. Your task is to reorder these stories in the following way: if you could only see one story,
which would it be? That story should be first. If you could only see two stories, which would they be?
Those should be the first and second story. In other words, we’d like you to reorder the stories in order
of how much you would want to see them, with the one you want to see most first.
To do this, you can click on the story and drag it into the right order. Here are some tips for making this
task easier:
To see more stories at once, zoom out in your browser
Drag stories to order them horizontally, with the best story on the far left of the screen and the
least best story at the far right of the screen
Bakeoff
For the bakeoff task, you will view a series of story pairs and will choose which story you would rather
see in your News Feed. In other words, you will “bake off” two stories against each other and pick the
one you would like to see more. Here’s what the tool looks like:
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
32/36
32
To do this, use the left and right arrows on your keyboard to toggle between the two stories. Then click
the “Save Selection” button or press enter to make your selection. If you click the right arrow, the story
on the right will be outlined in green. If you click the left arrow, the story on the left will be outlined in
green. The story outlined in green at the time you press “Save Selection” or Enter should be the story
you would rather see.
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
33/36
33
Printable Reference Materials
Work Flow Chart
One page Reference Guide for printing
Please print the one-page reference guide on the last page and keep this next to you while you do the
ratings. When not working on the ratings, please store this one-page reference guide in a safe place.
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
34/36
34
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
35/36
Rating 1 2 3 4 5
Person / Page Someone you really
don’t care about
Someone you don’t
care about
Someone you
somewhat care about
Someone you care
about
Someone you really
care about
Content
Content you are
really not interested
to see
Content you are not
interested in seeing
Content you are
somewhat interested
in seeing
Content that you are
interested in seeing
Content you are
really interested in
seeing
Impact
Really does not have
an impact on you /
has a negative impact
Does not have an
impact on you
Has a little impact on
youHas an impact on you
Has a significant
impact on you
Overall
Rating
Definitely do NOT
want to see in Feed
Do not want to see in
Feed
Don’t mind seeing in
Feed but wouldn’t
mind missing either
Want to see in FeedDefinitely want to see
in Feed
Rating 1 2 3 4 5
How much does this story make you feel
entertained?
Really not
entertainedNot entertained
A little
entertainedEntertained
Really
entertained
How much does this story help you feel
connected to friends/ family?
Really not
connectedNot connected
A little
connectedConnected
Really
connected
How much does this story help you feel
informed about the world?
Really not
informedNot informed A little informed Informed Really informed
Content Type
Friend/Family = Stories about people you know
personally that do not contain Public content
Public = Page posts, news articles, or other stories
that are about public topics, not about your
friends/family
Both = Stories where you gain information about a
friend AND the Public content shared
Type of Impact
You use this information
in discussions with people,
online or offline.
You gain information that
you want to have.
You have a (desired)
emotional reaction.
You talk about or share what you saw or read with
others in some way.
The information you learn is interesting, informative,
useful, valuable, or relevant in some way.
You feel delight, joy, excitement, laughter, empathy,
happiness or another desired emotion.
8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf
36/36