Near Neighbor Problem Made Fairmahabadi/slides/FairNN.pdfAll existing algorithms for this problem...

Near Neighbor Problem Made Fair

Sariel Har-PeledUIUC

Sepideh MahabadiTTIC

Nearest Neighbor Problems

• Nearest Neighbor: Given a set of objects, find the closest one to the query object.

• Near Neighbor: given a set of objects, find one that is close enough to the query object.

There are many applications of NN

Searching for the closest object

Near Neighbor

Dataset of 𝑛𝑛 points 𝑃𝑃 in a metric space, e.g. ℝ𝑑𝑑, and a parameter 𝑟𝑟

Near Neighbor

Dataset of 𝑛𝑛 points 𝑃𝑃 in a metric space, e.g. ℝ𝑑𝑑, and a parameter 𝑟𝑟A query point 𝑞𝑞 comes online

𝑞𝑞

𝑟𝑟

Near Neighbor

Goal: • Find a point 𝑝𝑝∗ in the 𝑟𝑟-neighborhood

𝑞𝑞𝑝𝑝∗

𝑟𝑟

Near Neighbor

Goal: • Find a point 𝑝𝑝∗ in the 𝑟𝑟-neighborhood• Do it in sub-linear time and small space

𝑞𝑞𝑝𝑝∗

𝑟𝑟

Near Neighbor

Goal: • Find a point 𝑝𝑝∗ in the 𝑟𝑟-neighborhood• Do it in sub-linear time and small space

All existing algorithms for this problem• Either space or query time depending exponentially on 𝑑𝑑• Or assume certain properties about the data, e.g., bounded

intrinsic dimension

𝑞𝑞𝑝𝑝∗

𝑟𝑟

Approximate Near Neighbor

Goal: • Find a point 𝑝𝑝∗ in the 𝑟𝑟-neighborhood• Do it in sub-linear time and small space• Approximate Near Neighbor

─ Report a point in distance c𝑟𝑟 for c > 1

𝑞𝑞𝑝𝑝

𝑟𝑟𝑐𝑐𝑟𝑟

Approximate Near Neighbor

Goal: • Find a point 𝑝𝑝∗ in the 𝑟𝑟-neighborhood• Do it in sub-linear time and small space• Approximate Near Neighbor

─ Report a point in distance c𝑟𝑟 for c > 1─ For Hamming (and Manhattan) query time is 𝑛𝑛𝑂𝑂(1/𝑐𝑐) [IM98] ─ and for Euclidean it is 𝑛𝑛𝑂𝑂( 1

𝑐𝑐2) [AI08]

𝑞𝑞

𝑟𝑟𝑐𝑐𝑟𝑟𝑝𝑝

Fair Near Neighbor

Sample a neighbor of the query uniformly at random

Individual fairness: every neighbor has the same chance of being reported. Remove the bias inherent in the NN data structure

Fair Near Neighbor

Sample a neighbor of the query uniformly at random

Individual fairness: every neighbor has the same chance of being reported. Remove the bias inherent in the NN data structure

Applications:Removing noise, k-NN classificationAnonymizing the dataCounting the neighborhood size

Fair Near Neighbor

Goal: • Return each point 𝑝𝑝 in the neighborhood of 𝑞𝑞 with uniform

probability• Do it in sub-linear time and small space

𝑞𝑞12

𝑟𝑟 12

Approximate Fair Near Neighbor

Goal of Approximate Fair NN─ Any point 𝑝𝑝 in 𝑁𝑁(𝑞𝑞, 𝑟𝑟) is reported with “almost uniform”

probability, i.e., 𝜆𝜆𝑞𝑞(𝑝𝑝) where

11 + 𝜖𝜖 𝑁𝑁 𝑞𝑞, 𝑟𝑟

≤ 𝜆𝜆𝑞𝑞(𝑝𝑝) ≤1 + 𝜖𝜖𝑁𝑁 𝑞𝑞, 𝑟𝑟

𝑞𝑞12

𝑟𝑟 12

Results on (1 + 𝜖𝜖)-Approximate Fair NN

𝑆𝑆𝐴𝐴𝐴𝐴𝐴𝐴 and 𝑇𝑇𝐴𝐴𝐴𝐴𝐴𝐴 are the space and query time of standard ANN

Domain Guarantee Space Query

Exact Neighborhood𝑁𝑁(𝑞𝑞, 𝑟𝑟)

w.h.p 𝑂𝑂(𝑆𝑆𝐴𝐴𝐴𝐴𝐴𝐴) �𝑂𝑂(𝑇𝑇𝐴𝐴𝐴𝐴𝐴𝐴 ⋅𝑁𝑁 𝑞𝑞, 𝑐𝑐𝑟𝑟𝑁𝑁 𝑞𝑞, 𝑟𝑟

𝑞𝑞

𝑟𝑟

Approximate neighborhood: a set 𝑆𝑆 such that 𝑁𝑁 𝑞𝑞, 𝑟𝑟 ⊆ 𝑆𝑆 ⊆ 𝑁𝑁 𝑞𝑞, 𝑐𝑐𝑟𝑟

Approximate Neighborhood𝑁𝑁 𝑞𝑞, 𝑟𝑟 ⊆ 𝑆𝑆 ⊆ 𝑁𝑁(𝑞𝑞, 𝑐𝑐𝑟𝑟)

In expectation 𝑂𝑂(𝑆𝑆𝐴𝐴𝐴𝐴𝐴𝐴) �𝑂𝑂(𝑇𝑇𝐴𝐴𝐴𝐴𝐴𝐴)

𝑞𝑞

𝑟𝑟𝑞𝑞

𝑟𝑟𝑐𝑐𝑟𝑟

Dependence on 𝜖𝜖 is O(log(1𝜖𝜖))

Experiments

Recent paper [Aumuller, Pagh, Silvestry’19] defining the same notion

Locality Sensitive Hashing (LSH)One of the main approaches to solve the Nearest Neighbor problems

Hashing scheme s.t. close points have higher probability of collision than far points

Locality Sensitive Hashing (LSH)

Hashing scheme s.t. close points have higher probability of collision than far pointsHash functions: 𝑔𝑔1 , … ,𝑔𝑔𝐿𝐿

• 𝑔𝑔𝑖𝑖 is an independently chosen hash function

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

Hashing scheme s.t. close points have higher probability of collision than far pointsHash functions: 𝑔𝑔1 , … ,𝑔𝑔𝐿𝐿

• 𝑔𝑔𝑖𝑖 is an independently chosen hash function

If 𝑝𝑝 − 𝑝𝑝′ ≤ 𝑟𝑟 , they collide w.p. ≥ 𝑃𝑃ℎ𝑖𝑖𝑖𝑖ℎIf 𝑝𝑝 − 𝑝𝑝′ ≥ 𝑐𝑐𝑟𝑟 , they collide w.p. ≤ 𝑃𝑃𝑙𝑙𝑙𝑙𝑙𝑙

For 𝑃𝑃ℎ𝑖𝑖𝑖𝑖ℎ ≥ 𝑃𝑃𝑙𝑙𝑙𝑙𝑙𝑙

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

Retrieval: [Indyk, Motwani’98]• The union of the query buckets is roughly

the neighborhood of 𝑞𝑞

• ⋃𝑖𝑖 𝐵𝐵𝑖𝑖 𝑔𝑔𝑖𝑖 𝑞𝑞 is roughly the neighborhood

𝑞𝑞

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

• How to report a uniformly random neighbor from union of these buckets?

𝑞𝑞

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

• How to report a uniformly random neighbor from union of these buckets?

• Collecting all points might take 𝑂𝑂(𝑛𝑛) time

𝑞𝑞

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

Approaches

How to output a random neighbor from ⋃𝑖𝑖 𝐵𝐵𝑖𝑖 𝑔𝑔𝑖𝑖 𝑞𝑞 :

1. Choose a uniformly random bucket2. Choose a uniformly random point in the

bucket

Approach 1: Uniform/Uniform

𝑞𝑞

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

1. Choose a random bucket proportional to its size

2. Choose a random point in the bucket

Approach 2: Weighted/Uniform

𝑞𝑞

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

2. Choose a random point in the bucket Each point 𝑝𝑝 in the neighborhood is picked

w.p. proportional to its degree 𝑑𝑑𝑝𝑝

Approach 2: Weighted/Uniform

𝑞𝑞

Number of buckets that 𝒑𝒑appears in

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

3. Keep 𝑝𝑝 with probability 1𝑑𝑑𝑝𝑝

, o.w. repeat

Approach 3: Optimal

𝑞𝑞

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

, o.w. repeat

Uniform probability

𝑞𝑞

Approach 3: Optimal

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

, o.w. repeat

Uniform probability Need to spend 𝑂𝑂(𝐿𝐿) to find the degree

𝑞𝑞

Approach 3: Optimal

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

, o.w. repeat

Uniform probability Need to spend 𝑂𝑂(𝐿𝐿) to find the degree Might need 𝑂𝑂 𝑑𝑑𝑚𝑚𝑚𝑚𝑚𝑚 = 𝑂𝑂(𝐿𝐿) samples Total time is 𝑂𝑂(𝐿𝐿2)

𝑞𝑞

Approach 3: Optimal

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

Approximate the degree 𝑑𝑑𝑝𝑝Sample 𝑂𝑂( 𝐿𝐿

𝑑𝑑𝑝𝑝⋅𝜖𝜖2) buckets out of 𝐿𝐿 buckets to (1 + 𝜖𝜖)-approximate the degree.

Still if the degree is low this takes 𝑂𝑂(𝐿𝐿) samples.

Case 1: Small degree 𝒅𝒅𝒑𝒑:• More samples are required to estimate• Reject with lower probability -> Fewer queries of this type

Case 2: Large degree 𝒅𝒅𝒑𝒑:• Fewer samples are required to estimate• Reject with higher probability -> More queries of this type

Case 1: Small degree 𝒅𝒅𝒑𝒑:• More samples are required to estimate• Reject with lower probability -> Fewer queries of this type

Case 2: Large degree 𝒅𝒅𝒑𝒑:• Fewer samples are required to estimate• Reject with higher probability -> More queries of this type

This decreases 𝑂𝑂(𝐿𝐿2) runtime to �𝑂𝑂(𝐿𝐿) Large dependency on 𝜖𝜖 of the form 𝑂𝑂( 1

𝜖𝜖2)

Via a different sampling approach we show how to reduce the dependency to logarithmic 𝑂𝑂(log 1

𝜖𝜖).

Experiments

Setup• Take MNIST as the data set• Ask a query several times and compute the empirical distribution of the

neighbors.• Compute the statistical distance of the empirical distribution to the uniform

distribution

Experiments

Setup• Take MNIST as the data set• Ask a query several times and compute the empirical distribution of the

neighbors.• Compute the statistical distance of the empirical distribution to the uniform

distributionComparison• Our algorithm performs 2.5 times worse than the optimal algorithm, but the

other two perform 7 and 10 times worse than the optimal.• Four times faster than the optimal but 15 times slower than the other two

Conclusion

The dependence on the parameter 𝑛𝑛 matches the standard Nearest Neighbor.We get an independent near neighbor each time we draw a sample. More generally the approach works for sampling form a sub-

collection of sets.

Conclusion

collection of sets.Open Problem:

o Finding the optimal dependency on the density parameter: 𝐴𝐴 𝑞𝑞,𝑐𝑐𝑐𝑐𝐴𝐴 𝑞𝑞,𝑐𝑐

Conclusion

ThanksQuestions?

collection of sets.Open Problem:

o Finding the optimal dependency on the density parameter: 𝐴𝐴 𝑞𝑞,𝑐𝑐𝑐𝑐𝐴𝐴 𝑞𝑞,𝑐𝑐

Near Neighbor Problem Made Fairmahabadi/slides/FairNN.pdfAll existing algorithms for this problem...

Documents

SOLVING A SUPPLY CHAIN MANAGEMENT PROBLEM TO NEAR ... · Optimisation to yield near optimal results across a large set of problem ... holds a major importance since primary transports

1 Nearest Neighbor Search Problem: what's the nearest restaurant to my hotel?

The Traveling Salesman Problem Nearest-Neighbor Algorithmpeople.hsc.edu/faculty-staff/robbk/Math111/Lectures... · The Traveling Salesman Problem Nearest-Neighbor Algorithm Lecture

arxiv.orgarXiv:1807.07527v1 [cs.DS] 19 Jul 2018 OPTIMAL LAS VEGAS APPROXIMATE NEAR NEIGHBORS IN ℓp ALEXANDER WEI Harvard University Abstract. We show that approximate near neighbor

FOCUSFOCUS ONON FAITH€¦ · 10.11.2018 · Love thy Neighbor: thy homeless neighbor, thy gay neighbor, thy Muslim neighbor, thy black neighbor, thy immigrant neighbor, thy Jewish

Quantifying the Noisy Neighbor Problem in Openstack the Noisy Neighbor Problem in Openstack ... ZeroStack has an OpenStack client in Golang ... sequential/random pattern

Anomalously strong near-neighbor attraction in doped 1D

1 Near-Neighbor Search Applications Matrix Formulation Minhashing

SIMP: Accurate and Efﬁcient Near Neighbor Search in High ...€¦ · nearest neighbor queries. Solutions supporting a set of query ranges suffer from prohibitive space cost. There

Neighbor 2 Neighbor Lead Flip Book

Machine Learning is based on Near Neighbor Set(s), NNS. Clustering, even density based, identifies near neighbor cores 1 st (round NNS s, about a center)

Guest Lecture on N-body problems George Biros The ... · The nearest neighbor problem • ⇢-Nearest neighbor: Given q 2 Rd, ﬁnd all r i such that d(r i,q)

Near-Neighbor Methods in Random Preference Completionhomepages.rpi.edu/~liua6/docs/NN_presentation.pdf · 2/27/2019 · Near-Neighbor Methods in Random Preference Completion Settings

SOFSEM 2007 Weighted Nearest Neighbor Algorithms for the Graph Exploration Problem on Cycles

Approximate Nearest Neighbor Search in High Dimensionspeople.csail.mit.edu/indyk/icm18.pdfNearest Neighbor Search •Best match problem [Minsky-Papert’69], Post office problem [Knuth’73]

A ppr oximate Near est Subspace Sear ch with A pplications ...W e approach the nearest subspace problem by reducing it to the well explored nearest neighbor (NN) prob lem for points

The Skills Gap and the Near-Far Problem in Executive

Near Neighbor Search in High Dimensional Data (2)

Near-Optimal Discretization of the Brachistochrone Problem - CCRMA

Identification of Burkholderia pseudomallei Near-Neighbor ... · Charles Darwin University Identification of Burkholderia pseudomallei Near-Neighbor Species in the Northern Territory