Multi-Perspective, Simultaneous Embedding

Md Iqbal Hossain, Vahan Huroyan, Stephen Kobourov, Raymundo Navarrete

Fig. 1: Given several distances between a set of objects and matching projection planes, MPSE computes 3Dcoordinates for all the objects, such that when projected to the corresponding 2D planes the distances in each planematch the corresponding input distances. The input dataset is inspired by a sculpture “1, 2, 3” by James Hopkins.

Abstract— We describe MPSE: a Multi-Perspective Simultaneous Embedding method for visualizing high-dimensional data, basedon multiple pairwise distances between the data points. Specifically, MPSE computes positions for the points in 3D and providesdifferent views into the data by means of 2D projections (planes) that preserve each of the given distance matrices. We considertwo versions of the problem: fixed projections and variable projections. MPSE with fixed projections takes as input a set of pairwisedistance matrices defined on the data points, along with the same number of projections and embeds the points in 3D so that thepairwise distances are preserved in the given projections. MPSE with variable projections takes as input a set of pairwise distancematrices and embeds the points in 3D while also computing the appropriate projections that preserve the pairwise distances. Theproposed approach can be useful in multiple scenarios: from creating simultaneous embedding of multiple graphs on the same setof vertices, to reconstructing a 3D object from multiple 2D snapshots, to analyzing data from multiple points of view. We providea functional prototype of MPSE that is based on an adaptive and stochastic generalization of multi-dimensional scaling to multipledistances and multiple variable projections. We provide an extensive quantitative evaluation with datasets of different sizes and usingdifferent number of projections, as well as several examples that illustrate the quality of the resulting solutions.

Index Terms—Graph visualization, Dimensionality reduction, Multidimensional scaling, Mental map preservation.

1 Introduction

Given a high dimensional dataset, one of the main visualizationgoals is to produce an embedding in 2D or 3D Euclidean space,that suitably captures pairwise relationships among the repre-sented data. Dating back to the 1960s, a classical tool thatis widely used for both graphs and high dimensional datasetvisualization is Multidimensional Scaling (MDS), which aimsto preserve the distances between all pairs of datapoints ornodes/objects [43]. Dimensionality reduction is a more generalversion of this problem, aiming to project a given dataset to alower dimensional space. There are many popular algorithmsfor dimensionality reduction, from linear methods such as prin-cipal component analysis (PCA) [21], to non-linear methodssuch as t-SNE [32] which captures local neighborhoods andclusters and UMAP [34] which aims to capture both local andglobal structure.

We consider situations where instead of just one pairwisedistance function, the input is a set of pairwise distance func-tions on the same set of objects. The optimization goal is toplace the points in 3D so that the embedding simultaneouslypreserves the distances between the objects when projected tosome planes. For example, Fig. 2 shows different views of a 3D

• Md Iqbal Hossain and Stephen Kobourov are with the ComputerScience Department of The University of Arizona, E-mails:hossain@email.arizona.edu and kobourov@cs.arizona.edu.

• Vahan Huroyan and Raymundo Navarrete are with theDepartment of Mathematics of The University of Arizona,E-mails: vahanhuroyan@math.arizona.edu andraymundo@math.arizona.edu.

visualization of a network dataset with multiple attributes. Inthis visualization, when viewing the 3D coordinates from thecorrect perspective, the apparent distance between nodes rep-resents the true network distances for one of the attributes. Seesection 4.4 for details about the dataset and the visualization.

For the case of graph visualisation, consider a set of verticesV (e.g., researchers in one university) and several relationshipsdefined between them E1, E2, E3 (e.g. joint research publi-cations, joint research proposals, membership in different de-partments). We would like to compute a layout L in 3D aswell as 3 planes (P1, P2 and P3) such that when V is projectedonto plane P1 we see the graph G = (V,E1) so that distancesbetween vertices in the plane P1 correspond to the distancesdefined by E1. Similarly, when L is projected onto plane P2 wesee the graph G = (V,E2) so that distances between verticesin the plane P2 correspond to the distances defined by E2, andthe same for P3 and E3.

In both settings, this is a strict generalization of the clas-sical MDS problem, which can be seen as a special case withonly one pairwise distance function. Even for this special casethis problem is known to be difficult as the standard optimiza-tion approaches such as gradient descent are not guaranteedto converge to the global optimum. Nevertheless, in practice,when there is a clear structure in the given graph, MDS is of-ten likely to find a good local optimum and as we show in thispaper, the simultaneous optimization of our Multi-Perspective,Simultaneous Embedding (MPSE) produces good solutions.

A common approach for visualizing different relationshipson the same set of objects involves small multiples and oftensome mechanism (such as brushing and linking) to connect thesame objects in the different views, or morphing from one viewto the other. In contrast, MPSE produces one 3D layout and

arX

iv:1

909.

0648

5v3

[cs

.DS]

6 A

ug 2

020

Fig. 2: visualization of marriage and loan relations among prominent families in 15th-century Florence, viewed from differentangles. Each node represents one of the families. An edge is drawn between two families if at least one corresponding relationexisted between the families. The leftmost image shows the view that best captures pairwise distances of the marriage bondsbetween the families. The rightmost image shows the view that best captures pairwise distances of the loan bonds betweenthe families. The middle two images show other views of the 3D visualization. The embedding is computed using MPSE withvariable projections. A discussion about the quality of the embedding can be found in Section 4.4.

each of the different views is a 2D projection. In this way,MPSE attempts to balance the two main desirable qualitiesof good visualization of multiple relationships defined on thesame set of data: the readability of each individual view (typi-cally captured by a faithful embedding in 2D) and mental mappreservation (typically captured by keeping the objects in thesame position across different views). This cannot be accom-plished effectively in 2D as there simply is not enough space torealize more than one relationship well, while it becomes plau-sible in 3D, as our experiments indicate. MPSE can also allowone embedding to tell multiple stories, depending on the dif-ferent relationships in the data, captured by the correspondingprojections, as shown in Figures 2, 9 and 10. Finally, MPSEcan be used for reconstructing 3D structure from a collectionof 2D images, as shown in Figures 1, 3 and 7.

With the advent of virtual reality and augmented reality sys-tems, 3D visualization and 3D interaction with data in generaland graphs in particular are becoming a reality. Still, whenpresenting 3D results in a paper we are limited to showing 2Dsnapshots. The MPSE webpage http://mpse.arl.arizona.edu provides 3D visualizations with interactive examples, anda functional MPSE prototype.

We describe the MPSE method in detail. We consider twodifferent settings: one where the projection planes are given aspart of the input (e.g., the three sides of a 3D cube) and thesecond where computing the projection planes is part of theoptimization. Both settings have been implemented and workwell in practice. A fully functional prototype is available on thewebpage and source code is available on github. We providea description on the python implementation and provide sev-eral examples of MPSE embeddings with real-world examples.Finally, we provide quantitative evaluation of the scalabilityof the two variants with increasing number of data points andincreasing number of projections.

1.1 Previous work

We review work on visualizing multivariate and multilayer net-works, network layout algorithms, multidimensional scaling, si-multaneous embedding, and 3D reconstruction algorithms.

Multivariate network visualization. Multivariate [24] andmultilayer [18] graph visualization has received a great deal ofattention in the last couple of decades. Multi-label, multi-edge,multi-relational, multiplex, multi-modal and many other vari-ants are cleverly encapsulated by the general multilayer net-work definition of Kivela et al. [25].

Wattenberg’s PivotGraph [48] system can visualize and an-alyze multivariate graphs not using a global graph layout butrather a grid-based approach focusing on different relationshipsbetween node attributes. Semantic substrates [44] unfold mul-

tiple attributes of a graph, a pair of attributes at a time, us-ing two dimensions. Pretorius and van Wijk [40] describe aninteractive system that relies on clustering of both nodes andedges and interactive exploration using brushing and linking (aswell as parallel histograms) to show different graph attributes.GraphDice [7] is an interactive multivariate graph visualizationsystem that allows the selection of attributes from an overviewplot matrix. This results in a cross dimensional node-link plotfor every combination of attributes arranged as a matrix. Thissystem is built on the earlier ScatterDice system [15], whichprovides the ability to interactively explore multidimensionaldata using scatterplots. Transitions between the scatterplotswith one common component are performed as animated rota-tions in 3D space.

Different from our approach, most of the earlier methodsfocus on interactive visualizations of multivariate graphs wherechanging queries result in changing layouts and views. Theidea behind our MPSE approach is to produce one 3D layoutof the input graph and several projection planes, such that eachattribute corresponds to a projection plane in which geometricdistances correspond to the graph distances specified by theparticular attribute. The main advantage of this approach isthat it helps preserve the viewer’s 3D mental map, while alsocapturing different relationships in different projections of thesame underlying layout.

Network layout algorithms. Most basic network layoutsare obtained using force-directed algorithms. Also known asspring embedders, such algorithms calculate the layout of theunderlying graph using only information contained within thestructure of the graph itself, rather than relying on domain-specific knowledge [26]. Visual analytics systems for graphsusually focus on interaction [46]. MDS-like approaches todrawing graphs are exemplified in algorithms such as that ofKamada-Kawai [22] and Koren and Carmel [28]. Most com-monly used graph drawing systems, such as Graphviz [14],Pajek [6], Tulip [3] and Gephi [5], provide options to visual-ize graphs in 3D based on MDS-like optimization. Variantsof MDS are used in many graph layout systems, including[11, 16, 39, 47]. Other approaches to exploring layouts in 3Dinclude 3D hyperbolic and spherical spaces [12, 27, 35]. Noneof these earlier approaches however, provides a way to simul-taneously optimize different views for the same set of nodes.

Simultaneous embedding. This problem is also relatedto simultaneous graph embedding and matched drawings ofgraphs [8]. Specifically, in simultaneous geometric embeddingof two or more planar graphs requires planar straight-line draw-ings of each of the graphs, such that common vertices havethe same 2D coordinates in all drawings. This setting is veryrestrictive and solutions are guaranteed to exist for very re-

stricted type of input graphs, such as two paths [10], whileinstances with no solutions can be constructed from a pair oftrees [17] or even (path, tree) pairs [2]. Matched drawings re-quire straight-line drawings of the two or more input graphssuch that each common vertex has the same y-coordinate inall drawings. Pairs of trees and triples of cycles always havea matched drawing [19]. In general, instances with no solu-tion can be constructed from a pair of planar graphs, or evena (planar graph, tree) pair [13]. Note that matched pairs ofdrawings can be obtained from the MPSE embedding for everypair of graphs using the intersection line between the corre-sponding pairs of projection planes as the shared y-coordinatein the pair of matched drawings.

Multidimensional scaling. Consider the problem of recov-ering the positions (in Rp for some integer p > 0, typicallyp = 2) of a set of objects given their relative pairwise dis-tances. That is, given n objects with indices 1, 2, . . . , n, andgiven pairwise distances D = [Dij ]

ni,j=1, where Dij is the ob-

served distance between objects i and j, we wish to computepositions x1, x2, . . . , xn ∈ Rp such that ‖xi − xj‖ ≈ Dij for allobjects i and j. The matrix D may contain distances measuredin a higher dimensional Euclidean space, or a more generalmetric space, may be corrupted by noise, or may just repre-sent a measure of dissimilarity between the objects that doesnot come from a metric distance. (Metric) multidimensionalscaling (MDS) is a well known dimensionality reduction anddata visualization technique that can tackle this problem. Itsgoal is to find an embedding X = [x1, x2, . . . , xn] ∈ Rp of the nobjects so that their pairwise distances agree with the distanceor dissimilarity matrix D, by minimizing the stress function

σ2(X;D) :=∑i>j

(Dij −‖xi − xj‖)2 . (1)

Minimizing the stress function (1) is typically accomplished us-ing (stochastic) gradient descent [9] or stress majorization [16].

Note that the original formulation of multidimensional scal-ing is non-metric MDS. The problem was first studied in thenon-metric setting by Shepard [43] and Kruskal [30]. Non-metric MDS recovers structure from measures of similarity,based on the assumption of a reproducible ordering betweenthe distances, rather than relying on the exact distances.

Multi-view multidimensional scaling. Given a single pair-wise distance or dissimilarity matrix D, MDS aims to find anembedding X that minimizes MDS stress (1). In some appli-cations, data is collected from multiple sources, resulting inmultiple dissimilarity matrices. The following question arises:Given K dissimilarity matrices D = {D1,D2, . . . ,DK} of thesame n objects, how to construct a single embedding X thatbest represents all of the dissimilarities simultaneously? Theanswer to this question depends heavily on how the differentdissimilarity matrices are related to each other.

In the simplest of cases, the different dissimilarity matricesmay correspond to different noisy measurements of the samepairwise relationship. In this case, it may be possible to es-timate the true dissimilarity matrix D. For example, if noisein the observed dissimilarity matrices Dk can be assumed tobe independent and identically distributed (i.i.d.) random vari-ables, then the average (D1 +D2 + · · ·+DK)/K is an unbiasedestimate of D. It would then be possible to construct an MDSembedding using the estimate of D. Another alternative is toconstruct an embedding directly, by finding an embedding Xthat best approximates all of the pairwise dissimilarities simul-taneously, for example, by minimizing the functional

K∑k=1

σ2(X;Dk) =

K∑k=1

∑i>j

(Dkij −‖xi − xj‖

)2

.

Bai et. al [4] propose finding the embedding X thatminimizes the Multi-View Multidimensional Scaling (MVMD)stress function

SMV (X, α;D) =

K∑k=1

αγk∑i<j

(Dkij −‖xi − xj‖)2,

where the weights α = [α1, α1, . . . , αK ] are subject to the con-straints

K∑k=1

αk = 1, 0 ≤ αk ≤ 1,

and γ > 1 is a fixed parameter. Here the objective is to findan embedding X that minimizes a weighted sum of the MDSstress (1) for the different dissimilarity matrices. The param-eter γ balances between just finding the embedding that pro-duces the smallest single stress σ2(X;Dk) and assigning equalweights to all of the individual stress values. This idea, alongwith similar multi-view generalizations of other visualizationalgorithms, are implemented by Kanaan et al. [23].

In these approaches, the general assumption is that the dif-ferent dissimilarity matrices correspond to different views ofthe same relationship in the data. But what if the differentdissimilarity matrices truly measure different relationships inthe data? In this case, an embedding of the data whose corre-sponding pairwise distances try to agree with all of the pairwisedissimilarities is not useful. The multi-perspective simultane-ous embedding problem that we propose to solve with MPSEis different, as we ask whether it is possible to embed the dataso that different perspectives P k(X) of the embedding X canvisualize the different dissimilarities Dk simultaneously.

3D Reconstruction. Our problem is also related to 3D re-construction from a collection of 2D images. This problem hasbeen widely studied in different settings, including reconstruct-ing the underlying real 3D structure from large collections of2D photos [45]. More restricted variants are even closer to oursetting [29, 41]. Note however, that in our problem we have aconstant number of inputs (distance matrices or graphs) andthe projections we anticipate can be fixed or computed as a partof the optimization. Our proposed problem and its solution canbe viewed also as a step of the structure-from-motion (SfM)problem [36]. The SfM problem is one of the central problemsin computer vision which aims to recover the 3D structure ofa scene from a set of its projected 2D images. We relate ourproblem to the SfM problem by assuming some fixed pointsin the 3D scene which can be estimated on the corresponding2D projections via some feature extraction algorithm such asSIFT [31]. Once these points are estimated, we can measurethe pairwise distance matrices for these points and apply theMPSE algorithm to find the locations of these points in 3D.

1.2 Our Contribution

The main contribution in this paper is the introduction of themulti-perspective simultaneous embedding problem and a gen-eralization of MDS to multiple distance matrices to solve it.This is at the core of the proposed MPSE method for visualiz-ing the same dataset/graph in 3D from several different views,each of which captures a different set of distances/relationships.We consider two main variants: one in which each of the dif-ferent distances/relationships is associated with a specific 2Dprojection plane, and the other where computing the projectionplanes is also part of the optimization. Extensive quantitativeexperiments show that the method scales well with increasingnumber of data points and increasing number of projections.

2 Multi-Perspective, Simultaneous Embedding

Our proposed MPSE algorithm is a generalization of the stan-dard (metric) MDS problem. The setting of MDS and its corre-sponding optimization function, which is called stress function

is discussed in Section 1.1. We remark that if the minimumof the MDS stress function is zero, then the objects can bepositioned in Rp so that their pairwise distances exactly rep-resent the pairwise dissimilarities of D. If the minimum of theMDS stress function is nonzero but not too large, the mini-mizer of the MDS stress function provides an approximate wayto visualize the dissimilarities in Rp.

Suppose now that instead of a single pairwise dissimilar-ity matrix D, we observe multiple pairwise dissimilarities ma-trices D = {D1,D2, . . . ,DK} for the same set of n objects.It is natural to ask if MDS can be generalized to producean embedding so that the dissimilarities D1,D2, . . . ,DK canbe visualized by the relative positions of the objects. If itis assumed that the dissimilarities in D are no more thanapproximations of some hidden true dissimilarity matricesD1true,D

2true, . . . ,D

Ktrue, then simple generalizations of MDS

exist, as described in section 1.1. In all of these generalizations,an embedding X is produced so that the distances ‖xi − xj‖are as close as possible to all of the dissimilarity measurementsD1ij ,D

2ij , . . . ,D

Kij . Nonetheless, this is an undesirable assump-

tion if the goal is to visualize multiple distinct relationshipsbetween the same objects. In particular, the dissimilarity mea-surements D1

ij ,D2ij , . . . ,D

Kij can be so different that no em-

bedding X can produce pairwise distances ‖xi − xj‖ that helpvisualize all of these relations in a meaningful way.

Our generalization of MDS is inspired by the problem of3D reconstruction from multiple 2D images. We first explainour motivation with an example based on a sculpture of JamesHopkins, which illustrates how different the same object canlook when observed from a different viewpoints. Dependingon the direction from which the viewer sees the sculpture, theviewer will see a different digit ’1’ or ’2’ or ’3’. For our ex-periments, we recreated the “1, 2, 3” sculpture and uniformlyat random fixed a set of points in the three dimensions thatproduces the same visual effect when plotted. The set of pointsforms a figure ’1’ when viewed from the x-axis, it forms a figure’2’ when viewed 60 degrees towards the y-axis, and finally as afigure ’3’ when viewed another 120 degrees towards the y-axis.Suppose now that the 3D structure is unknown, and the goal isto construct it using only the information from these 3 images.That is, we know the 3 distance matrices D1,D2,D3 measur-ing the distances between the same set of keypoints in each ofthese images, and the goal is to find an embedding X so that

Dkij ≈ ‖P k(xi)− P k(xj)‖, k = 1, 2, 3, i, j = 1, 2, . . . , n,

where P 1, P 2, P 3 are the projections that map keypoints in 3Dto their corresponding images. For this example one can takethe following projection matrices:

P1 =

[0 1 00 0 1

],P2 =

[√3

212

00 0 1

],P3 =

[√3

2− 1

20

0 0 1

].

(2)We can ask two questions: 1) given the dissimilarity matricesand the projections, can we recover the embedding X? 2) giventhe dissimilarity matrices, can we recover the projection matri-ces and the embedding X? Clearly the latter question is harderto answer. In both cases, we suggest a natural generalizationof the MDS stress function

3∑k=1

∑i>j

(Dkij − ‖P k(xi)− P k(xj)‖

)2

.

This can be naturally generalized to any number of distance(dissimilarity) matrices and projections. Even if the multipledissimilarity measures at hand are not related to each other insuch a geometric way, a similar idea may still provide a usefulvisualization of the multiple relations involved.

Motivated by these examples, we generalize the MDS stressfunction in the following way: Given n objects with L distinct

dissimilarity matrices Dl, the multi-perspective MDS stressfunction is defined as

S(X,P;D) =

L∑k=1

σ2(P (k)(X);D(k))

=

L∑k=1

∑i>j

(D

(k)ij −‖P

(k)(xi)− P (k)(xj)‖)2

.

(3)

Here, P = [P (1), P (2), . . . , P (K)], where P k : Rp → Rq

are the mappings (projections). We write P (k)(X) =

[P (k)(x1), P (k)(x2, . . . , P(k)(xn)]. We mainly focus on the spe-

cial case of linear orthogonal mappings, which correspond toorthogonal projections. There are two different ways in whichthe stress function (3) can be minimized: 1) Given a fixed setof mappings P l, we find a minimizer X ∈ Rn×p; 2) we finda pair (X,P) minimizing the stress, where X ∈ Rp×n and themappings P k for 1 ≤ k ≤ L are restricted to a class of functions(such as linear orthogonal transformations from Rp to Rq).

2.1 MPSE with Fixed Perspectives

We first consider minimization of the multi-perspective MDSstress function (3) with respect to the embedding X only:

minimizexi∈Rp

S(X,P;D) (4)

In this setup, the perspective mappings P (1), . . . , P (K) : Rp →Rq are predetermined and fixed. We assume that these map-pings are differentiable, so that a solution to (4) may be at-tainable using a gradient descent scheme.

We remark that the objective function for MPSE with fixedprojections in (4) is differentiable. Since the multi-perspectivesMDS stress function (3) is generally non-convex, minimizationto a global minimum is not guaranteed by a vanilla imple-mentation of gradient descent. In addition, computing the fullgradient of (3) at each iteration is expensive for large data setsand might make the algorithm infeasible. Thus coming up witha fast and accurate solution is one of the main contributions inthis paper. In section 3, we describe a stochastic gradient de-scent scheme with adaptive learning rate for efficient solutionto this problem.

2.2 MPSE with Varying Projections

We again consider minimization of the multi-perspective MDSstress function (3), but we no longer assume predeterminedand fixed perspective mappings P. Our goal is then to findboth the embedding X and the perspective mappings P thatbest capture the given distance matrices D. In order to dothis, a suitable parametric family O of C1(Rp,Rq) perspectivemappings must be defined. The optimization problem becomes

minimizeX∈Rp×n, P (k)∈O

S(X,P;D) (5)

For ease of exposition, we only consider the set of orthogonalprojections for the family of perspective mappings. Section 6.By O3×2 we denote the family of orthogonal projections fromR3 to R2. Then, each perspective mapping can be parameter-ized by an orthogonal p×q matrix Q. The perspective mappingP (Q) is given by x 7→ Qx. Note that the perspective mappingsvary smoothly with respect to changes in the parameters, inthe sense that the map Q 7→ P (Q)(x) is C1 for every x ∈ Rp.The optimization problem (5) can then be rewritten as

minimizeX∈Rp×n, Q(k)∈O3×2

n∑k=1

σ2(Q(k) (X) ;D) (6)

We remark that the objective function of MPSE with varyingprojections in (6) is also differentiable. However, since the set

O3×2 of 3 by 2 orthogonal matrices is not a subspace of R3×2,minimizing (6) with respect to Ql cannot be accomplished viagradient descent. Instead, we make use of projected gradientdescent, where Ql is updated by first moving towards the direc-tion of steepest descent, and then projecting back onto the setO3×2. If A ∈ R3×2 matrix, then the projection of A onto O3×2

is the matrix Π(A) ∈ O3×2 that minimizes ‖A − Q‖F amongall Q ∈ O3×2. There is a simple way to compute Π(A): ifUΣV T is the reduced singular value decomposition of A, thenΠ(A) = UV T .

2.3 Some Extensions

The MDS stress function can be extended to general graphstructures. That is, we do not require that a pairwise dissimi-larity relation exists between every pair of nodes. Moreover, weassume we are given a weighted pairwise relation, where someedges contribute to the stress function more than the others.To make stress values more meaningful, it is common to workwith a normalized MDS stress function. The normalized MDSstress function is

σ2(X,D) =

∑(i,j)∈E wij (Dij −‖(xi)− (xj)‖)2∑

(i,j)∈E wij(Dij)2. (7)

where E is the set of edges in the graph for which a dissimilarityrelation exits and wij is the weight corresponding to edge (i, j).

We also generalize MPSE to weighted graphs, where the nor-malized MPSE stress function becomes

S2(X,P;D) =1

K

K∑k=1

σ2(P (k)(X),D(k)), (8)

where P(k)(X) = [P(k)(x1),P(k)(x2), . . . ,P(k)(xN )] for eachk = 1, 2, . . . ,K.

3 Algorithms

In this section we discuss our proposed algorithms for solvingthe optimization problems for MPSE with fixed projections (4)and MPSE with varying projections (5). In our experiments,we found that convergence to a global minimum of (3) is un-likely using basic gradient descent algorithms. We found thata combination of smart initialization and stochastic gradientdescent with adaptive learning rate usually produces better re-sults than the vanilla version of the gradient descent. The ben-efits of using stochastic gradient are twofold: first, it is fasterthan the vanilla version of the gradient descent, and second, asobserved by others [49], it helps avoid local minima.

Given a dissimilarity matrix D, let ξ ∼ Ξ(D, c) be a randomvariable returning a dissimilarity matrix ξ, where for each (i, j),1 ≤ i, j ≤ n we include Dij with probability c and otherwise setξij = 0. Let σ2(X, ξ) and ∇Xσ

2(X, ξ) be unbiased estimatesof the MDS stress and its gradient at X. Then, the multi-perspective MDS stress (3) is approximated by

S2ξ (X,P;D) = S2(X,P, ξ)

where ξ = [ξ1, . . . , ξK ] and ξ(k) ∼ Ξ(D(k), c). The gradients∇XS

2ξ (X,P;D) and ∇Q(k)S2

ξ (X,P;D) are similarly computed.Note that all these can be computed simultaneously with onepass through the edge list.

We use an adaptive gradient descent framework from [33]with the following adaptive scheme for the learning rate:

µX(Xt,Xt−1, ξt) =(Xt−Xt−1)T (∇XS

2ξt(Xt−∇XS

2ξT

(Xt−1))∥∥∥∇XS2ξt

(Xt)−∇XS2ξt

(Xt−1)∥∥∥2

(9)The adaptive learning rate for the projection matrices µQ(t)

is computed in a similar fashion.

We summarize the resulting algorithm for MPSE with fixedprojections(4) in Algorithm 1. We remark that the problem isnon-convex and there is no guarantee for convergence. How-ever, our extensive experiments show that with smart initial-ization, or with multiple runs of the algorithm (with differentrandom initialization), the algorithm consistently finds goodsolutions. The idea behind the smart initialization is that eventhough the problem is non-convex, around the global optimumthe problem is well-behaved and so starting close to the globaloptimum we can apply gradient descent.

Algorithm 1 MPSE with fixed projections (problem (4))

Input: Dissimilarity matrices: D = {D1, . . . ,DK}, initial em-bedding X0, initial learning rate µ0, stochastic constant c,iteration number T .

for t = 1, 2, . . . , T doξt ∼ Ξ(D, c)Xt = Xt−1−µt−1∇XS

2ξt(Xt,P;D)

µt = µ(Xt−1,Xt, ξt)end for

Output: XT

Next, we describe the algorithm for MPSE with variableprojections (5) and summarize it in Algorithm 2. We remarkthat this problem is more complex then MPSE with fixed pro-jections. Variable projections offer more degrees of freedom,at the expense however of non-convex optimization and non-trivial local optima. Extensive experiments in this setting alsoshow that the algorithm works well in practice.

Algorithm 2 MPSE with varying perspectives (problem (5))

Input: Dissimilarity matrices: D = {D1, . . . ,DK}, initialembedding and perspective parameters X0 and Q0, initiallearning rates µX, and µQ, stochastic constant c, iterationnumber T .for t = 1, 2, . . . , T do

ξt ∼ Ξ(D, c)Compute ∇S2

ξ (Xt−1,Qt−1) and ∇S2ξ (Xt,Qt)

Compute µt.Xt = Xt−1−µX,0∇XS

2ξ (X,Qt).

Qt = Π(Qt − µQt∇QS2ξ )

µXt = µ(Xt−1,Xt, ξt)

µQt = µ(Qt−1,Qt, ξt)end for

Output: Xfinal, Qfinal.

Algorithm 3 summarizes our proposed smart initialization ofX0 and Q0. We have found that a preliminary computationof the optimal ’combined’ MDS embedding of the dissimilarityrelations works best. Finally, we remark that the standardpractice of multiple runs with different random initializationalso helps avoid local minima.

4 Experimental Evaluation

In this section we experimentally evaluate the proposed MPSEalgorithms: with fixed projections and with variable projec-tions. Since the problem that we study is new, there are noalgorithms to compare against; instead, we create benchmarkdatasets that we believe can demonstrate and test the MPSEalgorithms. For each dataset, we first describe how it was cre-ated and show the outputs of MPSE. Note that Algorithm 2,similar to the regular MDS, computes results invariant to globalrotation/reflection and there is no way to recover the exact ori-entation of the embedding without additional information.

Fig. 3: The output of Algorithm 2 on the distance matrices of the One-Two-Three dataset form Fig. 4. The first subfigureshows the projection which recovers digit 2, the third one corresponds to digit 1, and the last one corresponds to digit 3. Thesecond and fourth shows the rotational transition in 3D between 2 and 1 and between 1 and 3. The final MPSE stress for theOne-Two-Three dataset with varying projections (Algorithm 2) is 0.07, with projection-wise stress values of 0.04, 0.08 and 0.08.

Algorithm 3 Initialization for Algorithm (2)

Input: Dissimilarity set D, random initial embedding and per-spective parameters X0 and Q0, initial learning rates µX ,and µQ, stochastic constant c, iteration number T .

D2ij = d1

d2K

∑Kk=1(D

(k)ij )2 . Combine dissimilarities.

for t = 1, 2, . . . , T doξt ∼ Ξ(D, c)Xt = Xt−1−µt−1∇XS

2ξt(Xt,P;D)

µt = µ(Xt−1,Xt, ξt)end for . Compute MDS embedding of D on Rd1 :X = Xt.for t = 1, 2, . . . , T do

ξt ∼ Ξ(D, c)Qt = Qt−1−µt−1∇QS

2ξt(Xt,Qt;D)

µt = µ(Xt−1,Xt, ξt)end for . Compute optimal perspective parameters givenembedding X:X = Xt.

Output: Xfinal, Qfinal.

4.1 One-Two-Three Dataset

In the introduction we motivated the MPSE problem using thesculpture “1, 2, 3” by James Hopkins. To generate the datasetfor Fig. 1 we attempted to reverse-engineer the sculpture asfollows: We created 2D visualizations of the 3 projections bycreating images of the digits 1, 2 and 3 and sampled 1000 pointsfrom each; see Fig. 4.

Fig. 4: One-Two-Three Dataset: each subfigure contains 1000points sampled from the original images of digits 1, 2 and 3.The points are labeled from top to bottom to preserve theircorrect positions.

Then we computed the 2D distances for each set of pointsand fed those as input to the MPSE algorithm, aiming to re-cover the 3D figure of the sculpture. The goal is to see whetherit is possible to place 1000 points in 3D so that it would looklike digits 1, 2 and 3 when seen from different viewpoints.

We first run Algorithm 1 for the distance matrices createdfrom the One-Two-Three dataset; see Fig. 4. For the algo-rithm we used the following parameters: maximum number ofiterations T = 100, fixed projections P1,P2 and P3 from (2),starting learning rate µ0 = 1 and stochastic constant c = 0.01with random initialization. The results are shown in Fig. 1 and

indicate that the algorithm successfully placed the points in 3Dsuch that one can see digits 1, 2 and 3 (see the first, third andfifth subfigures of Fig. 1). The quality of the MPSE embeddingin Fig. 1 is also quantified with low stress values: overall stress0.001, with projection-wise stress values of 0.0009, 0.0008 and0.0011.

Next, we run Algorithm 2 for the same One-Two-Threedataset, with the following parameters: maximum number ofiterations T = 100, the starting learning rate µ0 = 1 andstochastic constant c = 0.01 with smart initialization describedin Algorithm 3. The results are shown in Fig. 3. We remarkthat for the MPSE with varying projections the results arecorrect, up to a global rotation and reflection. The algorithmagain successfully placed the points in 3d such that one cansee digits 1, 2 and 3 (see the first, third and fifth subfigures ofFig. 3).

4.2 Circle-Square Dataset

The Circle-Square dataset is also an example of 3D share re-construction from 2D. For this dataset we construct 2D imagesof a unit square and a circle with radius 1. From these figureswe sample 100 points; see Fig. 5. We create the correspondingtwo distance matrices and run the MPSE algorithms.

Fig. 5: Circle-Square dataset: the input is 100 points sampledfrom a square and a circle.

We first run Algorithm 1 for the distance matrices createdfrom the Circle-Square dataset; see Fig. 5. For the algorithmwe used the following parameters: maximum number of itera-tions T = 500, fixed projections P1, and P2 from (2), startinglearning rate µ0 = 1 and stochastic constant c = 0.01 withrandom initialization. The results are shown in the first rowof Fig. 7. The algorithm successfully placed the points in 3Dso that one can see a circle and a square from different planes(first and last subfigures). The second and third subfiguresshow the rotational transition in 3D between the circle and thesquare.

Next, we run Algorithm 2 for the same Circle-Square datasetwith the following parameters: maximum number of iterationsT = 100, the starting learning rate µ0 = 1 and stochasticconstant c = 0.01 with smart initialization described in Algo-rithm 3. The results are shown in the second row of Fig. 7. Aswe can see the algorithm again successfully placed the points

Fig. 6: The output of Algorithm 2 on the Clustering dataset. The first subfigure shows the 3D embedding and the remainingones correspond to the 3 projections. To make it easier to see the 3 different types of clusters captured here, in all subfigureswe use green/red to distinguish between datapoints from different clusters of the first type, circle/square glyphs for clusters ofthe second type, and filled/empty glyphs for clusters of the third type. We manually add the separating lines to highlight theseparations realized in the 3 planes. The final MPSE stress for the Clustering dataset with varying projections (Algorithm 2) is0.60, with projection-wise stress values of 0.52, 0.47 and 0.77.

Fig. 7: The output of Algorithms 1 and 2 applied on the Circle-Square dataset. The first row show the result of Algorithm 1and the second row shows the result of Algorithm 2. For bothrows, the first column captures the recovered circle and the lastrow captures the recovered square. The second and third rowsshow the rotational transition in 3D between the circle and thesquare, revealing its 3D structure. The final MPSE stress forthe circle-square dataset with fixed projections (Algorithm 1)is0.046, with projection-wise stress values of 0.045 and 0.047.The final MPSE stress for the circle-square dataset with vary-ing projections (Algorithm 2) is 0.044, with projection-wisestress values of 0.044 and 0.045.

in 3D so that one can see a circle and a square from differentdirections (first and last subfigures).

4.3 Clusters Dataset

One of the many applications of dimensionality reduction isto preprocess the dataset by reducing its dimension and thenapply a clustering/classification algorithm. The setting wherewe want to test whether our proposed algorithm preserves theclusters of a given dataset is the following: We want to visual-ize a dataset in 3D such that different 2D projections capturedifferent clusters (say, determined by considering different at-tributes of the data). For this purpose, we fix n = 200 andcreate 3 datasets in 2D each having 2 well separated clusters of100 datapoints each. Our goal is to apply Algorithms 1 and 2

Fig. 8: The input to the Clusters dataset: each subfigure showsan example of a dataset of 200 points with 2 well separatedclusters of 100 points.

and check whether the corresponding 3D embeddings preservethe clusters and if so what the corresponding projections looklike. We apply Algorithm 2 with T = 100, the distance ma-trices corresponding to datapoints of subfigures of Fig. 8 andrandom initialization. The results are shown in Fig. 6.

Remarkably, as shown in Fig. 6, the MPSE algorithmwith varying projections captured the special structure of thedataset (three different pairs of clusters defined on the samedatapoints) and embedded them in 3D so that the 3 differentseparations can be seen from the corresponding projections.

4.4 15th-century Florentine Families Dataset

We now consider a network dataset that captures social rela-tionships between prominent families in Renaissance Florence(one of the largest and richest cities in 15th century Europe)[38, 42]. This dataset contains descriptions of social ties be-tween Florentine families during the years 1426-1434, a periodof historical significance that marks the rise to power of theMedici family.

The social ties are divided into categories such as ‘marriage’(number of marriages between two families) and ‘loan’ (numberof loans between two families). We can interpret this dataset asa network with multiple attributes. To visualize this network,we select a subset of the families such that the network struc-ture for the ‘marriage’ (1) and ‘loan’ (2) attributes each form aconnected graph. For each type of relation k ∈ {1, 2}, we definethe dissimilarity between two families as follows: for families

i and j, if there is a nonzero number n(k)ij of ties between the

two families, then the pairwise dissimilarity is initially given

by D(k)ij = 1/n

(k)ij ; afterwards, shortest path distances are com-

puted for all pairs of families using this initial set of dissimilar-ities, resulting in dissimilarities for every pair of families. Thepairwise weights used to produce the MDS (7) and MPSE (8)

embeddings are defined by w(k)ij = 1/D

(k)ij .

We compute a 2D MDS embedding based on marriage dis-tances and a 2D MDS embedding based on loan distances (in-dependent of each other). We then compute a 3D simulta-neous embedding based on both marriage and loan distancesusing MPSE with variable projections with the following pa-rameters: maximum number of iterations T = 300, startinglearning rate µ0 = 1, stochastic constant c = 0.1, and initialembedding from Algorithm 3.

The first row of images in Fig. 10 shows the two 2D MDS em-beddings and the second row of images shows two projectionsof the 3D MPSE embedding. The first column corresponds tomarriages and the second column corresponds to loans. Thestress value for each embedding is the standard normalizedMDS stress (7). Hence, the MDS and MPSE values are directlycomparable to each other. Note that the individual normalizedMDS stress for both perspectives in the MPSE embedding are

MPSE embedding: gender view

Female

Male

MPSE embedding: education view

Lower secondary

Secondary

Incomplete higher

Higher

MPSE embedding: income view

Fig. 9: The output of Algorithm 2 on the credit card dataset. The first subfigure shows the 3D embedding and the remainingones correspond to the 3 projections on gender, education and income views, respectively. In all figures, high color intensityrepresents high income, color represents gender, and glyph shapes represent education levels. The final MPSE stress for thecredit card dataset with varying projections is 0.40, with projection-wise stress values of 0.28, 0.29 and 0.55.

MediciStrozzi

MDS embedding of marriage network

Medici

Strozzi

MDS embedding of loan network

MediciStrozzi

MPSE embedding: marriage view

Medici

Strozzi

MPSE embedding: loan view

Fig. 10: MDS and MPSE embeddings of the Florentine familiesdataset based on marriages and loans. The first row showsthe two 2D MDS embeddings (obtained independently of eachother) and the second row shows two projections of the 3DMPSE embedding from Fig. 2. The color indicates the truegraph distance from the Medici family for each attribute withblue indicating close neighbors and yellow indicating distantones (as described in the text). The MDS stress values are 0.70for the marriage distances and 0.65 for the loan distances. Thetotal MPSE stress is 0.78, with projection-wise stress values of0.75 for marriage perspective and 0.81 for the loan perspective.

close to the optimal normalized MDS stress values (computedindividually for each attribute). This indicates that it is possi-ble to visualize both attributes simultaneously with the MPSEmethod, without significantly increasing stress values.

The colors in Fig. 10 indicate the true graph distance fromthe Medici family, with respect to marriage relations (the firstcolumn) and loan relations (the second column). A “perfect”embedding would result in dark blue colors near the Medicifamily and yellow for the nodes farthest away (with gradualblue-to-yellow transition in between). Note that we can obe-serve this phenomenon both in the individual MDS embeddingsand in the MPSE embeddings in Fig. 10. Different views of the3D MPSE embedding are shown in Fig. 2, with the view cor-responding to the optimal MPSE perspective of the marriageattribute on the left and the view corresponding to the optimalMPSE perspective of the loan attribute on the right.

A bit of historical background can help us see some interest-ing results in our visualizations. The Strozzi family had beenFlorence’s richest one, but ended up exiled and in ruin after

the Medici took control of Florence’s government in 1434. TheMPSE embedding, illustrated in Fig. 2 and Fig. 10, allows usa glimpse in this story. The Medici family occupies a centrallocation in both the individual MDS and in the MPSE embed-dings. The Strozzi family occupies a similarly central positionin the marriage perspectives, but not in the loan perspectives.This reflects the nature of their political power at the time,as both families has strong ties to other prominent families inFlorence, but the Medici family was doing better financiallyby that time. The better positioning of the Medici family isnicely illustrated in the MPSE 3D embedding in Fig. 2 in a waythat cannot be illustrated by the MDS embeddings: the Medicifamily has a central position in the 3D embedding (with an av-erage pairwise distance of 2.87), whereas the Strozzi family liescloser to the periphery (with an average pairwise distance of4.05). Even though their position in the marriage network wasfirm, their position in the overall network was weaker.

4.5 Credit Card Application Dataset

Another real-world, high-dimensional dataset comes fromanonymized credit card applications [1]. The dataset con-tains information about customers of a bank (gender, educationlevel, income, etc.). We randomly selected 1000 customers andfocused on three parameters: annual income, gender, and levelof education. The goal is to use MPSE to embed the dataset in3D so that three different projections show us patterns based onthe corresponding parameters. Since MPSE requires numericinput we modify some of the parameters a bit. For gender wemap male and female to 0 and 1. For education level, we maplower secondary, secondary, incomplete higher and higher edu-cation to 0, 1, 2, 3. To avoid giving more importance to someof the features, each set of features is then normalized so thatthe scale of all MPSE perspectives is the same.

We compute an MPSE variable projections embedding (Al-gorithm 2), after initialing it with Algorithm 3, using the fol-lowing parameters: maximum number of iterations T = 200,starting learning rate µ0 = 1, and stochastic constant c =0.004. The results are shown in Fig. 9. The first subfigureshows one view of the resulting 3D embedding, and the remain-ing subfigures show the projections of the embedded datasetonto the gender, education level, and income views.

The 3D embedding seems to capture well the different per-spectives, shown in the 3 different projections. The gender viewshows clearly separated clusters of male (orange) and female(blue color) customers. The education view groups customerswith similar education levels (see the groups of squares, cir-cles, and triangles). The income view attempts to “sort” thecustomers by income (high on the top, low on the bottom inthis figure), as captured in the change of color intensity fromthe top to the bottom. This view also shows us that educa-tion level and income are correlated (circles at the top), andthat men have higher income than women (the orange clustersare higher than the blue clusters). Indeed the actual averages

for male and female customers in this sample of size 1000 are$195000 and $152000, respectively.

5 Testing the scalability of Algorithms 1 and 2

In this section we test whether Algorithms 1 and 2 scale oncethe number of datapoints and the number of projections in-crease. For this purpose we create random datasets, that iswe fix the number of datapoints n and sample n points uni-formly from a solid ball with radius 1 in 3D. Next, we fix thenumber number of projections K and generate K random pro-jections Q1, . . . ,QK of this data onto R2. From these pro-jected datasets we generate the corresponding distance matri-ces D1, . . . ,DK . The input for Algorithm 1 is D1, . . . ,DK andQ1, . . . ,QK and for Algorithm 2 the input is D1, . . . ,DK . Weconsider the following 2 experiments for Algorithm 1 and Algo-rithm 2: The first experiment fixes the number of projectionsto be K = 3 and varies the number of datapoints n between100 and 2000 in increments of 100. We fix the max numberof iterations T = 100 for the first experiment and computethe average time, and the number of times that the algorithmsuccessfully finds the global minimum over 10 instances.

0 500 1000 1500 2000

Number of datapoints

0

2

4

6

8

10

Avera

ge tim

e (

seconds)

Fixed projections

Variable Projections

500 1000 1500 2000

Number of datapoints

65

70

75

80

85

90

95

100

105

Avera

ge s

uccess p

robabili

ty

Fixed projections

Variable projections

0 5 10 15 20

Number of Projections

0

2

4

6

8

10

Avera

ge tim

e (

seconds)

Fixed projections

Variable projections

0 5 10 15 20

Number of projections

65

70

75

80

85

90

95

100

105

Avera

ge s

uccess p

robabili

ty

Fixed projections

Variable projections

Fig. 11: Summary of the results for the scalability tests. Thefirst row corresponds to the experiments where we fix the num-ber of projections K = 3 and vary number of datapoints be-tween 200 to 2000 with increments of 100. The second rowcorresponds to the experiments where we fix the number ofdatapoints n = 200 and vary the number of projections Kfrom 1 to 19 with increments of 1. The first column reportsthe average running time of the algorithms over 10 instancesand the second row shows the success rates of the algorithms.

The second experiment fixes n = 200 and varies the num-ber of projections K between 1 and 20 in increments of 1. Wefix the max number of iterations T = 1000 and compute theaverage time and number of times that the algorithm success-fully finds the global minumum over 10 instances. The resultsare reported in Fig. 11. We remark that, as shown in Fig. 11,the running time of Algorithms 1 and 2 increases linearly withthe increase of datapoints with Algorithms 2. Similarly, therunning time of both algorithms increases linearly with the in-crease of the number of projections. The second column ofFig. 11 shows that the increase in the number of datapointsand the increase of number of projections does not effect thesuccess rate of both algorithms.

Fig. 12 shows the computation history for Algorithm 1 whenapplied to the “1, 2, 3” dataset which resulted in the imagesshown in Fig. 1. The left image shows multi-perspective MDSstress at each iteration. The right image shows the behavior of

parameters (normalized gradient size, learning rate, and nor-malized step size) during the execution of the algorithm.

Fig. 12: Computation history of Algorithm 1 for the “1, 2, 3”data shown in Fig. 1. The left image shows the decay of to-tal multi-perspective MDS stress at each iteration. The rightimage shows the behavior of parameters (normalized gradientsize, learning rate, and normalized step size) during the exe-cution of the algorithm. We used the following parameters forAlgorithm 1: initial learning rate µ0 = 1, stochastic constantc = 0.02, and random initial embedding. The final MPSE stressof 0.001 was reached after 58 iterations, with projection-wiseMDS stress values of 0.0009, 0.0008 and 0.0011.

6 Computation of the gradients

In this section, we present and derive the formulas for the gradi-ent functions of the multi-perspective MDS stress function (3).Our purpose is to assist the reader who wishes to implementthese algorithms. Otherwise this section can be skipped.

For a given n × n dissimilarity matrix D = [Dij ]ni,j=1, the

classical (metric) MDS stress function σ2 : RT×n → R is givenby (1). Assume the stress function for Y = [y1, y2, . . . , yn] ∈RT×n instead of X, we plan to use X for the MDSE.

For a function f : RT×n → R, we denote by Dyif(Y ) : RT →R the derivative operator of yi 7→ f(Y ) at Y . Since

‖(yi + δ)− yj‖2 = ‖yi − yj‖2 + 2(yi − yj)T δ +O(‖δ‖2),

it follows that

Dyi‖yi − yj‖2 = 2(yi − yj)T

and

Dyi‖yi − yj‖ = Dyi√‖yi − yj‖2 =

(yi − yj)T

‖yi − yj‖,

so that

Dyiσ2(Y;D) = 2

∑j 6=i

‖yi − yj‖ −Dij

‖yi − yj‖(yi − yj)T . (10)

The gradient ∇yiσ2(Y;D) ∈ RT is given by

∇yiσ2(Y;D) = 2

∑j 6=i

‖yi − yj‖ −Dij

‖yi − yj‖(yi − yj). (11)

Suppose now that Y = [P (x1), P (x2), . . . , P (xn)], wherex1, . . . , xn ∈ RS and P : RS → RT is differentiable. We writeX = [x1, x2, . . . , xn] and Y = Y (X,P ). We have

Dxiσ2(Y (X,P );D) = Dyiσ

2(Y ;D)DP (xi),

where DP (x) ∈ L(RS ,RT ) is the derivative of P at x ∈ RS . Itfollows that

∇xiσ2(Y (X,P );D) = ∇P (xi)∇yiσ

2(Y ;D), (12)

where ∇P (x) = [∇P1(x), . . . ,∇PT (x)] ∈ RS×T .Suppose now K distinct n × n dissimilarity matrices

D(1),D(2), . . . ,D(K) are given for the same n objects. We writeD = {D(1),D(2), . . . ,D(K)}. We wish to explore how each

dissimilarity matrix D(k) can arise from a perspective map-ping P (k) : Rp → Rq. In our first algorithm, the perspectivemappings are known, and the only requirement is that eachmapping P (k) is differentiable. In the second algorithm, themappings are no longer known, but are assumed to be param-eterized.

The multi-perspective MDS stress function S : RS×n → R isgiven by

S(X;P,D) =

K∑k=1

σ2(Y(X,P(k));D(k)). (13)

so that

DxiS(X,P,D) =

K∑k=1

Dyiσ2(Y;D(k))DP(k)(xi)

and therefore

∇xiS(X;P,D) =

K∑k=1

∇P(k)(xi)∇yiσ2(Y(X;P(k));D(k)).

(14)

If the perspective mappings are linear, so that P (k)(x) =

Q(k)x for some T × S matrix Q(k), then Y(X, P (k)) = Q(k) X

and ∇P (k)(x) = Q(k) for any x. The gradients of S reduce to

∇xiS(X;P,D) =

K∑k=1

Q(k)∇yiσ2(Q(k) X;D(k)) (15)

In our second approach to optimizing (3), it is not assumedthat the perspective mappings are known, but instead thatthese belong to some parametric family of C1(RS ,RT ) maps.We assume that the set of allowed parameters Q ⊂ Rr is openin Rr. For each q ∈ Q, we write P (q) : RS → RT . Furthermore,the mappings P (q) are assumed to be differentiable with respectto q, in the sense that the function q 7→ P (q)(x) is differentiablefor all x ∈ RS .

For a given x ∈ RS and q ∈ Q, we write DqP(q)(x) ∈

L(RS ,RT ) as the derivative of q 7→ P (q)(x). Consider the func-

tion q 7→ σ2(Y(X,P(q));D). The derivative is

Dqσ2(Y (X,P(q));D) =

n∑i=1

Dyiσ2(Y(X,P(q));D)Dq P(q)(xi)

and its gradient is

∇qσ2(Y(X,P(q));D) =

n∑i=1

∇qP (q)(xi)∇yiσ2(Y(X,P(q));D)

(16)

For linear correspondences q 7→ P (q)(x), DqP(q)(x) ∈

L(Rr,RT ) is given by DqP(·)(x) 7→ P (·)(x) and therefore

∇q P(q)(x) = [P(e1)(x), . . . ,P(er)(x)]T

where e1, . . . , er is the standard basis for Rr.

For r = ST , for a given q ∈ Rr, we rewrite q as a T by Smatrix Q, so that P (q)(x) = Qx. we have

∇Qσ2(Y(X,P(q));D) =

n∑i=1

∇yiσ2(QX;D)xTi (17)

This can be rewritten as

∇Qσ2(Y(X,P(q));D) = ∇Y σ2(QX;D)XT (18)

For stochastic gradient descent schemes, an approximation to11 can be constructed as follows: the n objects are divided intobatches; if object i belongs to batch b(i) with size |b(i)|, thenthe approximate gradient is given by

[∇σ2(Y;D)]i = 2n

|b(i)|∑

j∈b(i), j 6=i

‖yi − yj‖ −Dij

‖yi − yj‖(yi − yj).

(19)

7 Conclusions and Future Work

We described a generalization of MDS which can be used tosimultaneously optimize multiple distance functions defined onthe same set of objects. The result is an embedding in 3D spacewith a set of given or computed projections that show the dif-ferent views. This approach has applications for visualizingabstract data and multivariate networks. It would be interest-ing to experiment and test whether MPSE can compete withcurrent techniques for 3D reconstruction from 2D images. Forexample, one could combine the MPSE approach with a multi-way matching algorithm [20, 37] and a feature extraction algo-rithm [31] to automatically detect points of interests in each im-age, match them, and place them in 3D to recover the 3D figure.The MPSE webpage http://mpse.arl.arizona.edu provides3D visualizations with interactive examples, and a functionalMPSE prototype; https://github.com/enggiqbal/MPSE pro-vides the source code.

Acknowledgements

This research was supported in part by National Science Foun-dation grants CCF–1740858 and DMS-1839274.

We thank the participants in the “Visualizing DynamicGraphs in VR” working group at Shonan Seminar 131 “Im-mersive Analytics for Network and Trail Sets Data Analysis”:David Auber, Peter Eades, Seokhee Hong, Hiroshi Hosobe,Stephen Kobourov, Kwan-Liu Ma, and Ken Wakita.

References

[1] Credit Card Fraud Detection. https://www.kaggle.com/c/

home-credit-default-risk. Accessed: 2020-07-20.[2] P. Angelini, M. Geyer, M. Kaufmann, and D. Neuwirth. On a

tree and a path with no geometric simultaneous embedding.In International Symposium on Graph Drawing, pp. 38–49.Springer, 2010.

[3] D. Auber, D. Archambault, R. Bourqui, M. Delest, J. Dubois,A. Lambert, P. Mary, M. Mathiaut, G. Melancon, B. Pinaud,et al. Tulip 5, 2017.

[4] S. Bai, X. Bai, L. J. Latecki, and Q. Tian. Multidimensionalscaling on multiple input distance matrices. In Thirty-FirstAAAI Conference on Artificial Intelligence, 2017.

[5] M. Bastian, S. Heymann, M. Jacomy, et al. Gephi: anopen source software for exploring and manipulating networks.ICWSM, 8:361–362, 2009.

[6] V. Batagelj and A. Mrvar. Pajek-program for large networkanalysis. Connections, 21(2):47–57, 1998.

[7] A. Bezerianos, F. Chevalier, P. Dragicevic, N. Elmqvist, andJ.-D. Fekete. Graphdice: A system for exploring multivariatesocial networks. In Computer Graphics Forum, vol. 29, pp.863–872. Wiley Online Library, 2010.

[8] T. Blasius, S. G. Kobourov, and I. Rutter. Simultaneous embed-ding of planar graphs. In R. Tamassia, ed., Handbook of GraphDrawing and Visualization, pp. 349–381. CRC Press, 2013.

[9] L. Bottou. Large-scale machine learning with stochastic gradi-ent descent. In Proceedings of COMPSTAT’2010, pp. 177–186.Springer, 2010.

[10] P. Brass, E. Cenek, C. A. Duncan, A. Efrat, C. Erten, D. P.Ismailescu, S. G. Kobourov, A. Lubiw, and J. S. Mitchell. Onsimultaneous planar graph embeddings. Computational Geom-etry, 36(2):117–130, 2007.

[11] L. Chen and A. Buja. Local multidimensional scaling fornonlinear dimension reduction, graph drawing, and proxim-ity analysis. Journal of the American Statistical Association,104(485):209–219, 2009.

[12] I. F. Cruz and J. P. Twarog. 3d graph drawing with simulatedannealing. In International Symposium on Graph Drawing, pp.162–165. Springer, 1995.

[13] E. Di Giacomo, W. Didimo, M. J. van Kreveld, G. Liotta, andB. Speckmann. Matched drawings of planar graphs. J. GraphAlgorithms Appl., 13(3):423–445, 2009.

[14] J. Ellson, E. R. Gansner, E. Koutsofios, S. C. North, andG. Woodhull. Graphviz - open source graph drawing tools. InGraph Drawing, pp. 483–484, 2001.

[15] N. Elmqvist, P. Dragicevic, and J.-D. Fekete. Rolling the dice:Multidimensional visual exploration using scatterplot matrixnavigation. IEEE transactions on Visualization and ComputerGraphics, 14(6):1539–1148, 2008.

[16] E. R. Gansner, Y. Koren, and S. North. Graph drawing by stressmajorization. In International Symposium on Graph Drawing,pp. 239–250. Springer, 2004.

[17] M. Geyer, M. Kaufmann, and I. Vrto. Two trees which areself–intersecting when drawn simultaneously. In InternationalSymposium on Graph Drawing, pp. 201–210. Springer, 2005.

[18] M. Ghoniem, F. Mcgee, G. Melancon, B. Otjacques, and B. Pin-aud. The state of the art in multilayer network visualization.arXiv preprint arXiv:1902.06815, 2019.

[19] L. Grilli, S.-H. Hong, G. Liotta, H. Meijer, and S. K. Wismath.Matched drawability of graph pairs and of graph triples. Com-putational Geometry, 43(6-7):611–634, 2010.

[20] V. Huroyan. Mathematical Formulations, Algorithms and The-ory for Big Data Problems. PhD thesis, University of Min-nesota, 2018.

[21] I. T. Jolliffe. Principal Component Analysis. Springer Series inStatistics. Springer, 1986. doi: 10.1007/978-1-4757-1904-8

[22] T. Kamada and S. Kawai. An algorithm for drawing generalundirected graphs. Inform. Process. Lett., 31:7–15, 1989.

[23] S. Kanaan-Izquierdo, A. Ziyatdinov, M. A. Burgueno, andA. Perera-Lluna. Multiview: a software package for multiviewpattern recognition methods. Bioinform., 35(16):2877–2879,2019.

[24] A. Kerren, H. C. Purchase, and M. O. Ward. Introductionto multivariate network visualization. In Multivariate NetworkVisualization, pp. 1–9. Springer, 2014.

[25] M. Kivela, A. Arenas, M. Barthelemy, J. P. Gleeson, Y. Moreno,and M. A. Porter. Multilayer networks. Journal of complexnetworks, 2(3):203–271, 2014.

[26] S. G. Kobourov. Force-directed drawing algorithms. InR. Tamassia, ed., Handbook of Graph Drawing and Visualiza-tion, pp. 383–408. CRC Press, 2013.

[27] S. G. Kobourov and K. Wampler. Non-eeuclidean spring em-bedders. IEEE Transactions on Visualization and ComputerGraphics, 11(6):757–767, 2005.

[28] Y. Koren and L. Carmel. Visualization of labeled data usinglinear transformations. In IEEE Symposium on InformationVisualization 2003, pp. 121–128. IEEE, 2003.

[29] A. Koutsoudis, B. Vidmar, G. Ioannakis, F. Arnaoutoglou,G. Pavlidis, and C. Chamzas. Multi-image 3d reconstructiondata evaluation. Journal of Cultural Heritage, 15(1):73–79,2014.

[30] J. B. Kruskal. Multidimensional scaling by optimizing goodnessof fit to a nonmetric hypothesis. Psychometrika, 29(1):1–27,1964.

[31] D. G. Lowe. Object recognition from local scale-invariant fea-

tures. In Proceedings of the International Conference on Com-puter Vision, Kerkyra, Corfu, Greece, September 20-25, 1999,pp. 1150–1157. IEEE Computer Society, 1999. doi: 10.1109/ICCV.1999.790410

[32] L. v. d. Maaten and G. Hinton. Visualizing data using t-sne.Journal of machine learning research, 9(11):2579–2605, 2008.

[33] Y. Malitsky and K. Mishchenko. Adaptive gradient descentwithout descent. CoRR, abs/1910.09529, 2019.

[34] L. McInnes, J. Healy, N. Saul, and L. Großberger. UMAP: uni-form manifold approximation and projection. J. Open SourceSoftw., 3(29):861, 2018. doi: 10.21105/joss.00861

[35] T. Munzner. Exploring large graphs in 3d hyperbolic space.IEEE Computer Graphics and Applications, 18(4):18–23, 1998.

[36] O. Ozyesil, V. Voroninski, R. Basri, and A. Singer. A surveyof structure from motion. Acta Numer., 26:305–364, 2017. doi:10.1017/S096249291700006X

[37] D. Pachauri, R. Kondor, and V. Singh. Solving the multi-way matching problem by permutation synchronization. InAdvances in neural information processing systems, pp. 1860–1868, 2013.

[38] J. Padgett and C. Ansell. Robust action and the rise of themedici, 1400-1434. American Journal of Sociology, 98:1259–1319, 1993.

[39] C. Pich. Applications of multidimensional scaling to graphdrawing. PhD thesis, 2009.

[40] A. J. Pretorius and J. J. Van Wijk. Visual inspection of mul-tivariate graphs. In Computer Graphics Forum, vol. 27, pp.967–974. Wiley Online Library, 2008.

[41] Y. Queau, J. Melou, J.-D. Durou, and D. Cremers. Dense multi-view 3d-reconstruction without dense correspondences. arXivpreprint arXiv:1704.00337, 2017.

[42] T. Shafie. A multigraph approach to social network analysis.Journal of Social Structure, 16, 2015.

[43] R. N. Shepard. The analysis of proximities: multidimensionalscaling with an unknown distance function. Psychometrika,27(2):125–140, 1962.

[44] B. Shneiderman and A. Aris. Network visualization by semanticsubstrates. IEEE transactions on visualization and computergraphics, 12(5):733–740, 2006.

[45] N. Snavely, S. M. Seitz, and R. Szeliski. Modeling the worldfrom internet photo collections. International journal of com-puter vision, 80(2):189–210, 2008.

[46] T. Von Landesberger, A. Kuijper, T. Schreck, J. Kohlhammer,J. J. van Wijk, J.-D. Fekete, and D. W. Fellner. Visual analysisof large graphs: state-of-the-art and future research challenges.In Computer graphics forum, vol. 30, pp. 1719–1749. Wiley On-line Library, 2011.

[47] Y. Wang, Y. Wang, Y. Sun, L. Zhu, K. Lu, C.-W. Fu, M. Sedl-mair, O. Deussen, and B. Chen. Revisiting stress majorizationas a unified framework for interactive constrained graph visu-alization. IEEE transactions on visualization and computergraphics, 24(1):489–499, 2018.

[48] M. Wattenberg. Visual exploration of multivariate graphs. InProceedings of the SIGCHI conference on Human Factors incomputing systems, pp. 811–819. ACM, 2006.

[49] J. X. Zheng, S. Pawar, and D. F. Goodman. Graph drawing bystochastic gradient descent. IEEE transactions on visualizationand computer graphics, 25(9):2738–2748, 2018.