Analysis of strategic knowledge used in Back of
the Envelope Reasoning
Praveen ParitoshKen Forbus
QRG Meeting, 8th March 2005
Some examples
• How many K-8 school teachers are in the USA?• How much money is spent on newspapers in
USA per year?• What is the total annual gasoline consumption
by cars in US? • What is the annual cost of healthcare in USA?• How much power can a adult human generate? • What is Jason Kidd’s point per game for this
season?
Back of the Envelope Reasoning
• Numeric answers• Specificity-Resources tradeoff• Only thing you can do in many domains
– Environmental science, Biophysics, etc.– New, unfamiliar domains.
• Useful – Engineers, Scientists, Policymakers, everyone who
reads a newspaper
• Problem solving + feel for numbers [contrast first principles reasoning]
1. IncompletenessDomain theories incomplete in coverage.
2. ConcretenessKnowledge of concrete, specific situations (made use of by analogical reasoning) in addition to first-principles reasoning.
3. Highly experiential: Experience improves - ability to reason through similar scenarios. - intuitions for quantity: what is reasonable, high, low in a domain.
4. Focused reasoningTight reasoning, as opposed to maintaining ambiguity for completeness
5. Pervasively quantitativeReal-world actions require that estimates manifest as exact values.
Constraints guiding Common Sense QR
How many K-8 school teachers are in the USA?
Number of teachers = number of students / students per teacherNumber of students = population * fraction in the age range of K-8 students * fraction of kids who go to school
Number of students = 300 mil * 9/75 * 1Number of teachers = 40 mil / 25 = 1.6 mil
1.9 million, source: Statistical Abstracts, 2003.
How much money is spent on newspapers in USA per year?
Total money spent = Money spent per buyer * number of buyersAnnual expense per buyer = Units bought per year * cost per unit
Annual expense per buyer = 365 days * $0.75 = $250Number of buyers = 300 mil * ¼ = 75 mil Total money spent = 75 mil * $250 = $20 billion
$26 billion, source: Statistical abstracts, 2003.
Formalizing BoTE reasoning
Estimate parameter directly
Create estimation model
Find modeling strategy
Find values forparameters
in model
Use known valueif available
Estimate basedon similar situation
1
2
3
Feel for numbers
Problem solving
Problem Solving
• Representation– Of problems, strategies and domain knowledge.
• Retrieval– Of relevant knowledge and strategies
• Reasoning– Workspace: keeping track of progress made.– Agenda: figuring out what to do next.
BotE-Solver
• Representation– Domain knowledge: 1.2 million fact subset of Cycorp’s
CYC knowledge base. – Strategies: Suggestions.
• Retrieval– Pattern matching and backward chaining.
• Reasoning– Built on top of FIRE reasoning engine– AND/OR tree as workspace. – Difficulty ordered agenda.
An example suggestion
(defSuggestion HouseholdStrategyForCountingUnits :trigger (unitsTotal ?obj ?place ?time ?total-units) :test (ownedBy ?obj FamilyCohabitationUnit) :subgoals ((numberOfHouseholds ?place ?time ?num-households) (unitsPerHousehold ?obj ?units-per-household)) :result-step (evaluate ?total-units (TimesFn ?num-households ?units-per-household)))
HouseholdStrategyForCountingUnits
Total annual gasoline consumption
Total miles driven
Miles per Gallon
MileageStrategy
Number of cars
Miles driven per day
Annual miles driven per car
Number of households
Number of cars per house
PerUnitStrategy
PerDayForYearStrategy
Suggestion/OR-node
Goal/AND-node
AND/OR Decomposition
Analogical Estimation
• Feel for numbers– Build symbolic representations for numbers
• Large, expensive, upper class, etc. [CARVE, Paritosh, 2003]
– Analogical estimator: makes guesses for a numeric parameter based on analogy.
– Representations augmented with symbolic representation lead to more accurate estimates [Paritosh, submitted to CogSci05].
Analysis of Strategic Knowledge
• Claim: There is a core collection of strategic knowledge, specifically, seven strategies that capture most of back of the envelope reasoning.
• Source: – Strategies in Bote-Solver– Analysis of problems from Force and
Pressure, Rotation and Mechanics, Heat and Astronomy from Clifford Swartz’s Back-of-the-Envelope Physics.
Analysis of strategic knowledge
• Knowledge Level (Newell, 1982)– A description of what the system knows
• Heuristic Classification (Clancey, 1985)– “Forms of knowledge and reasoning patterns of familiar problem
solving situations and solutions, heuristically related.”– Applied Newell’s paradigm to ten expert systems of the era
(MYCIN, SACON, WINE, etc.)
Data
Heuristic matchSolution AbstractionsData abstractions
Solutions
BotE Strategies
• Abstract form: (Q O ?Value)
• Some examples discussed – Money spent on : newspapers sold every year
in the US– Number of : k-8 teachers in the US
• Mapping from a question to Q:O not unique.
BotE Strategies
(Q O ?Value) → {(Qi Oi ?Vi)}
• Such that ?Vi are already known or easier to estimate.
• Each strategy also contains the answers to the following questions:– When does it apply– How do we combine ?Vi’s to find ?value– How sure are we of our estimate of ?value
Object based strategies
Given Q:O:?V, transform it into Q:O1:?V1, Q:O2:?V2,… such that
?V = f(?V1, ?V2,…)• Mereology:
– {Oi} is the complete set of objects that share the non-overlapping extensive parameter Q.
• Weight of a basket of fruits = sum of weight of each fruit and the basket.
• Similarity: – O is similar to O1 => ?V = ?V1– O is similar to {Oi} => ?V = average({?Vi})
• Ontology: – If O is an instance of a collection C for which we know the
range/distribution of Q, or whose other instances have a known value for Q.
Kinds of strategies
• Object based: (Q O ?V) → {(Q Oi ?Vi)}
• Quantity based: (Q O ?V) → {(Qi O ?Vi)}
• System based: (Q O ?V) → {(Qi Oi ?Vi)}
1. Mereology
• Extensive: If Q is an extensive parameter,
• {Oi} is the complete set of objects that share the non-overlapping extensive parameter Q.
• Weight of a basket of fruits = sum of weight of each fruit and the basket.
Quantity based strategies
Given Q:O:?V, transform it into Q1:O:?V1, Q2:O:?V2,… such that
?V = f(?V1, ?V2,…) -- (2)
• Density: Rates, densities, averages along a dimension multiplied by the extent along that dimension.
• Physical laws: F = m * a. [Domain specific knowledge]• Scale-up: unit conversion.
• The composition function, f, in equation 2 has to meet dimensional constraints.
get-solution
get-solution(original-goal) if in-play?(original-goal) get-next-solution(original-goal) else if the agenda is empty then quit process-agenda
get-next-solutionget-next-solution(ao-node) if there are cached solutions at this node ;; cur-bmarker points to the current solution increment cur-bmarker(ao-node) return cached solution if goal-node?(ao-node) if ao-node has a child suggestion that is in-play get-next-solution(in-play-suggestion(ao-node)) ;; No in-play suggestion, if this goal has a younger ;; sibling that can give us more new bindings, re-instantiate ;; this node with those bindings. if younger-siblings(ao-node) bindings = get-next-solution(younger-sibling(ao-node)) if bindings found enqueue-on-agenda(re-instantiate-node(ao-node,bindings)) if suggestion-node?(ao-node) ;; seek downward solution = get-next-solution(eldest-child,ao-node) add-bindings(solution,ao-node) increment cur-bmarker(ao-node) return solution
propagate-bindingspropagate-bindings(goal-node) if elder-sibling?(goal-node) increment cur-bmarker(goal-node) next-node = instantiate-node(elder-sibling(goal-node), current-bindings(goal-node)) add-to-tree(next-node) enqueue-on-agenda(next-node) return ;; No elder sibling, so we can now see ;; if the parent node got solved if parent(goal-node) exists bindings = get-next-solution(goal-node) while we have bindings for goal-node result-bindings = do-result-step(parent(goal-node)) if result-bindings found add-bindings(result-bindings,parent(goal-node) propagate-bindings(parent(parent(goal-node))) bindings = get-next-solution(goal-node)
process-agenda
process-agenda(ao-node) if goal-node?(ao-node) solutions = ask(goal-node) ;; primitive problem solving step if solutions found propagate-bindings(goal-node) else suggestions = gather-suggestions(goal-node) if suggestions found enqueue-on-agenda(suggestions) else ;; no suggestions found, see if we can ;; re-instiate this node if younger-siblings(goal-node) bindings = get-next-solution(younger-sibling(goal-node) if bindings found enqueue-on-agenda(re-instantiate-node(ao-node,bindings)) else update-failed (goal-node) if suggestion-node?(ao-node) ;; since the subgoals are ordered, add the first one to agenda enqueue-on-agenda(youngest-child(ao-node))
How much time would be saved by increasing the speed limit from 55 to 65 mph?
Total time spent driving = time spent driving per person * number of driversTime spent driving per person = Total distance driven per person /SpeedTotal distance driven per person = Distance per day * number of days driving
Number of drivers: assume one driver per household = 1/3 * population = 100 milTotal distance driven per person = 20 * 365 = 8000Time spent per person = 8000/55 = 150 hoursTime spent if it was 65 mph = 8000/65 = 120 hoursTime saved per person = 30 hrs
Total time saved = 3 billion hours = 300,000 years.
The Microsoft Army – How much does software for 500,000 computers cost?
Total cost on the software = cost of desktop software + server software + upgrades
Windows XP/Office CDW prices = 650Cost of desktop software = 500,000 * 650 = 325 mil Cost of upgrades = 500,000 * 200 = 100 mil
Assume 1 in 100 servers = 5000 serversExchange server = 600Windows 2003 server = 650SQL Server = 1000Cost of server software = 2250 * 5000 = 11.25 milCost of upgrades = 400 * 5000 = 2 mil
Total cost = 440 millionSoftmart, inc, PA got paid = 470 million
What is the annual cost of healthcare in USA?
Lets say everyone was insured.
Total cost of healthcare has to be less than insurance premiums, forthe insurance companies to stay in business.
Cost of healthcare = average insurance premium * population = 3,000 * 300 million ~ 1 trillion
[1.6 trillion last year. How does this work?]
How much tea is there in China?
Total tea = amount of tea consumed per day * stockpile in time
Tea consumed per day = tea consumed per person per cup * number of cups * population
1 billion people5 cups a day5 grams a cupLets say they stock an years supply25 billion grams = 25 million kilos per dayIn an year 365*25 million ~ 10 billion kilos
Numbers in symbolic knowledge representation
• Consider the Great black-bucked gull– Wingspan = Large– Wing-span = 0.272 sq. m.
• Numbers not handled right – Similarity: computing and making inferences– Retrieval– Generalization
CARVE: Symbolizing Quantity
• Don’t do it – Sorites– Context/Utterer sensitivity
• Dimensional partitions: Large and Small, based on distributional properties of the quantity.(isa Algeria
(HighValueContextualizedFn Area AfricanCountries))
• Structural partitions: Boiling point and Poverty line, denote changes of quality.
CARVE
Dimensional partitioning for each quantity
(isa Algeria (HighValueContextualizedFn Area AfricanCountries)...
Add these facts to original cases
Structural clustering using SEQL
S1 S2 S3Cj
Ci
Ci*
Quantity 1
L2L1
Analogical Estimator
(GrossDomesticProduct Brazil ?x)
• The value is known.
• Find an analogous case for which value is known.
• Find anything in the KB which might help me make an estimate.
Thesis and Evaluation
• More powerful and flexible back of the envelope reasoning can be done using these symbolic representations of quantity.
• Evaluation: – Corpus of problems with/without
representations generated by CARVE.
Numbers in News
• The cost of Mars Pathfinder mission was 270 million.
• 50 billion tax cut.
• 62,000 square feet.
An example
(defSuggestion VolumeFormulaForSphere (volumeOfObject ?obj ?vol) :test (shapeOfObject ?obj SphereShape) :subgoals ((radius ?obj ?radius)) :result-step (evaluate ?vol (TimesFn 4.187 ;;4/3*Pi (ExponentFn ?radius 3))))
SOLVE
• Suggestions based problem solver
• Represent problem solving progress in an and/or tree structure
• Incremental solution generation, get-next-solution
• Ordered subgoals
For More
The BotE website:
www.cs.northwestern.edu/~paritosh/bote/
Library of problems, solutions, strategies.
What is the total annual gasoline consumption by cars in US?Total consumption = Total miles driven/ miles per gallonTotal miles driven = Number of cars in the US * Miles driven per car per yearMiles driven per car per year = Miles driven per day * 365If we say that every household owns a car, since some don’t and some might have more than one, then Number of cars in the US = number of households = population / average size of American household.
Now we have a model, and using the following numbers,Population ~ 300 million, Average size of household ~ 3, Daily miles driven ~ 20, Miles per gallon for a car ~ 20. We get an estimate of 36.5 billion gallons.
35 billion gallons, Statistical Abstracts, 2003.
Axioms for HouseholdStrategyForCountingUnits(ist-Asserted SuggestionMT (suggestionResultStep HouseholdStrategyForCountingUnits (evaluate ?total-units (TimesFn ?num-households ?units-per-household)))) (ist-Asserted SuggestionMT (suggestionSubgoals HouseholdStrategyForCountingUnits (TheList (numberOfHouseholds ?place ?time ?num-households) (unitsPerHousehold ?obj ?units-per-household)))) (ist-Asserted SuggestionMT (suggestionGoalForm HouseholdStrategyForCountingUnits (unitsTotal ?obj ?place ?time ?total-units))) (ist-Asserted SuggestionMT (implies (ownedBy ?obj FamilyCohabitationUnit) (suggestFor (unitsTotal ?obj ?place ?time ?total-units) HouseholdStrategyForCountingUnits))) (ist-Asserted SuggestionMT (isa HouseholdStrategyForCountingUnits Suggestion)) (suggestionResultStep HouseholdStrategyForCountingUnits (evaluate ?total-units (TimesFn ?num-households ?units-per-household))) (suggestionSubgoals HouseholdStrategyForCountingUnits (TheList (numberOfHouseholds ?place ?time ?num-households) (unitsPerHousehold ?obj ?units-per-household))) (suggestionGoalForm HouseholdStrategyForCountingUnits (unitsTotal ?obj ?place ?time ?total-units)) (implies (ownedBy ?obj FamilyCohabitationUnit) (suggestFor (unitsTotal ?obj ?place ?time ?total-units) HouseholdStrategyForCountingUnits)) (isa HouseholdStrategyForCountingUnits Suggestion)
Uniform representation of domain knowledge and strategies provides substrate for learning.
Workings of the Solver
Control Status Flags• SOLVED• FAILED• MOOT
– MOOT-VIA-FAILURE– MOOT-VIA-SUCCESS
Solution generation• Incremental• Two modes
– Agenda processing– IN-PLAY processing
Subgoals• Fully ordered
Outline
• Examples and motivation
• Worked out examples
• Implementation of BotE-Solver
• Results and current status• What next
(annualSales NewspaperCopy UnitedStatesOfAmerica (YearFn 2003) ?money)(defSuggestion PerBuyerStrategy (annualSales ?obj ?place ?time ?money) :subgoals ((annualExpensePerBuyer ?obj ?place ?time ?money-per-buyer) (numberOfBuyers ?obj ?place ?time ?number)) :result-step (evaluate ?money (TimesFn ?money-per-buyer ?number)))
(defSuggestion UnitaryStrategyForCost (annualExpensePerBuyer ?obj ?place ?time ?money-per-buyer) :subgoals ((annualUnitsBoughtPerBuyer ?obj ?place ?time ?units) (cost ?obj ?unit-cost)) :result-step (evaluate ?money-per-buyer (TimesFn ?units ?unit-cost)))
(defSuggestion FractionOfTotalStrategy (numberOfBuyers ?obj ?place ?time ?number) :subgoals ((populationDuring ?place ?any-time ?total) (percentOfBuyers ?obj ?place ?percent)) :result-step (evaluate ?number (QuotientFn (TimesFn ?total ?percent) 100)))
How much money is spent on newspapers in USA per year?
(cardinality K-8SchoolTeacher ?numteachers)
(defSuggestion StudentsPerTeacherStrategy (cardinality K-8SchoolTeacher ?numteachers) :subgoals ((cardinality K-8SchoolStudent ?numstudents) (studentsPerTeacher K-8School ?perteacher)) :result-step (evaluate ?numteachers (TimesFun ?numstudents ?perteacher))) (defSuggestion UniformAgeDistributionStrategy (cardinality K-8SchoolStudent ?numstudents) :subgoals ((populationDuring UnitedStatesOfAmerica (YearFn 1997) ?population) (minimumAge K-8SchoolStudent ?min) (maximumAge K-8SchoolStudent ?max) (lifeExpectancyForGroupInRegion UnitedStatesOfAmerica Person (YearsDuration ?life))) :result-step (evaluate ?numstudents (TimesFn (QuotientFn (DifferenceFn ?max ?min) ?life) ?population))))
How many K-8 school teachers are in the USA?
ProblemNumber
Problem, and its predicate calculus representation
Answer found by BotE-Solver, and comparison to a known answer if available
#Axioms
1 How many popcorn kernels would it take to fill in the 1890 Maple big classroom?(CountContained CS381ClassRoom Popcorn ?number)
(?number . 1.343444e+7)BotE-Solver: 13 million Correct answer: not available!
30
2 How much money is spent on newspapers in the US?(annualSales NewspaperCopy UnitedStatesOfAmerica (YearFn 2003) ?money)
(?money . 2.1884363e+10)BotE-Solver: 21 billionCorrect answer: 26 billion
30
3 How many K-8 teachers are there in the US?(cardinality K-8SchoolTeacher ?numteachers)
(?numteachers . 1056454)BotE-Solver: 1.05 millionCorrect Answer: 1.9 million
20
4 What is the annual cost of healthcare in the US?(annualSales HealthCare UnitedStatesOfAmerica (YearFn 2003) ?money))
(?money . 799428834000)BotE-Solver: 0.8 trillionCorrect Answer: 1.6 trillion
12
Solved Problems - I
ProblemNumber
Problem, and its predicate calculus representation
Answer found by BotE-Solver, and comparison to a known answer if available
# Axioms
5 How many cars are bought per year in the US?(unitsBoughtPerYear Automobile UnitedStatesOfAmerica (YearFn 2003) ?num)
(?num . 8920000)BotE-Solver: 8.9 millionCorrect Answer: 8 million
30
6 What is the weight of garbage thrown away by American families each year?(annualProduction Garbage-Generic UnitedStatesOfAmerica (YearFn 2003) (Pound-UnitOfMass ?garbage-mass))
(?garbage-mass . 446000000)BotE-Solver: 446 million poundsCorrect answer: not available!
10
7 How many hotdogs are sold in a baseball season in Wrigley Field?(unitsSold HotDogSandwich WrigleyField BaseballSeason ?num-dogs)
(?num-dogs . 1600000)BotE-Solver: 1.6 millionCorrect answer: not available!
18
8 What is the total amount of gasoline consumption by cars in the US?(annualAutomobileGasConsumption UnitedStatesOfAmerica (YearFn 2003) (Gallon-US ?oil-consumption)))
(?oil-consumption . 32558000000)BotE-Solver: 32.6 billion gallons Correct Answer: 35 billion gallons
20
Solved Problems - II
Outline
• Examples and motivation
• Worked out examples
• Implementation of BotE-Solver
• Results and current status
• What next
Object based strategies
Given Q:O:?V, transform it into Q:O1:?V1, Q:O2:?V2,… such that
?V = f(?V1, ?V2,…)• Mereology:
– {Oi} is the complete set of objects that share the non-overlapping extensive parameter Q.
• Weight of a basket of fruits = sum of weight of each fruit and the basket.
• Similarity: – O is similar to O1 => ?V = ?V1– O is similar to {Oi} => ?V = average({?Vi})
• Ontology: – If O is an instance of a collection C for which we know the
range/distribution of Q, or whose other instances have a known value for Q.
Quantity based strategies
Given Q:O:?V, transform it into Q1:O:?V1, Q2:O:?V2,… such that
?V = f(?V1, ?V2,…) -- (2)
• Density: Rates, densities, averages along a dimension multiplied by the extent along that dimension.
• Physical laws: F = m * a. [Domain specific knowledge]• Scale-up: unit conversion.
• The composition function, f, in equation 2 has to meet dimensional constraints.