Research on Rahat Zamanhttps://rahatzamancse.netlify.app/categories/research/Recent content in Research on Rahat ZamanHugo -- gohugo.ioenrahatzamancse@gmail.com (Rahat Zaman)rahatzamancse@gmail.com (Rahat Zaman)©2024, All Rights ReservedThu, 21 Apr 2022 13:22:57 -0700monthlymonthly- Frailty Visualizationhttps://rahatzamancse.netlify.app/posts/frailty-viz/Thu, 21 Apr 2022 13:22:57 -0700rahatzamancse@gmail.com (Rahat Zaman)Fri, 17 Jun 2022 16:59:44 -0700https://rahatzamancse.netlify.app/posts/frailty-viz/<p>
<div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;">
<iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen="allowfullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/djRtdDh-ljk?autoplay=0&controls=1&end=0&loop=0&mute=0&start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"
></iframe>
</div>
<br />
<img src="https://rahatzamancse.netlify.app/images/post/frailty-viz/overview.png"><br />
<img src="https://rahatzamancse.netlify.app/images/post/frailty-viz/relationview.png"></p>
<div class="notices warning" data-title="Warning">
This is very WIP project under active development. For which, I have not included a lot of details and just kept everything as draft.
</div>
<br/>
<p>The visualization website is deployed at: <a href="http://river.cs.arizona.edu:1600/blob-viz">http://river.cs.arizona.edu:1600/blob-viz</a> (You need to be connected to University of Arizona Wifi or use the VPN provided by UofA).</p><p>
<div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;">
<iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen="allowfullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/djRtdDh-ljk?autoplay=0&controls=1&end=0&loop=0&mute=0&start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"
></iframe>
</div>
<br />
<img src="https://rahatzamancse.netlify.app/images/post/frailty-viz/overview.png"><br />
<img src="https://rahatzamancse.netlify.app/images/post/frailty-viz/relationview.png"></p>
<div class="notices warning" data-title="Warning">
This is very WIP project under active development. For which, I have not included a lot of details and just kept everything as draft.
</div>
<br/>
<p>The visualization website is deployed at: <a href="http://river.cs.arizona.edu:1600/blob-viz">http://river.cs.arizona.edu:1600/blob-viz</a> (You need to be connected to University of Arizona Wifi or use the VPN provided by UofA).</p>
<p>The GitHub public repository is here: <a href="https://github.com/enoriega/frailty_graph_viz">github.com/enoriega/frailty_graph_viz</a></p>
<p>An initial wishlist document that can be shared publicly can help understand the “Research” part <a href="https://docs.google.com/document/d/19CMeM9H-43MJDEW8B9ev0XY8BU8sF-7YlegWrRlpPxY/edit?usp=sharing">here</a></p>Rahat Zamanfeatured imagevisualizationresearch
- 3D Reconstruction with MPSEhttps://rahatzamancse.netlify.app/posts/mpse-mds/Sat, 13 Nov 2021 13:22:57 -0700rahatzamancse@gmail.com (Rahat Zaman)Fri, 17 Jun 2022 16:59:44 -0700https://rahatzamancse.netlify.app/posts/mpse-mds/<p><img src="https://rahatzamancse.netlify.app/images/post/mpse-mds/contribution.png"></p>
<p><div class="notices info" data-title="Info">
The following images are interactive!!!
</div><br />
<iframe src="https://rahatzamancse.netlify.app/others/post/mpse-mds/chair.html" style="border:none;height:500px;width:100%;" title="Chair"></iframe><br />
<iframe src="https://rahatzamancse.netlify.app/others/post/mpse-mds/desk.html" style="border:none;height:500px;width:100%;" title="Desk"></iframe></p>
<div class="notices info" data-title="Info">
All of the experiments done for this research are managed by neptune.ai which can be found in <a href="https://app.neptune.ai/rahatzamancse/MPSE-with-GAN/experiments?split=tbl&dash=charts&viewId=52255a62-06ba-4223-8940-c2d840d58d1e">this link</a>.
</div>
<h2 id="multi-perspective-simultaneous-embedding-with-mdshttpsarxivorgabs190906485"><a href="https://arxiv.org/abs/1909.06485">Multi-Perspective Simultaneous Embedding with MDS</a></h2>
<div class="notices info" data-title="Info">
A single page quick review of this paper can be found <a href="https://drive.google.com/file/d/1-dtJw2hgenwCvndu85WZotSLk4MvSD_d/view?usp=sharing">here</a>.
</div>
<p>MPSE stands for Multi-Perspective Simultaneous Embedding which is a method proposed by <a href="https://www2.cs.arizona.edu/~kobourov/">Dr. Stephen Kobourov</a> et. al. for visualizing high-dimensional data, based on multiple pairwise distances between the data points. Specifically, MPSE computes positions for the points in 3D and provides different views into the data by means of 2D projections (planes) that preserve each of the given distance matrices. There are two versions of it: fixed projections and variable projections. MPSE with fixed projections takes as input a set of pairwise distance matrices defined on the data points, along with the same number of projections and embeds the points in 3D so that the pairwise distances are preserved in the given projections. MPSE with variable projections takes as input a set of pairwise distance matrices and embeds the points in 3D while also computing the appropriate projections that preserve the pairwise distances. The proposed approach can be useful in multiple scenarios: from creating simultaneous embedding of multiple graphs on the same set of vertices, to reconstructing a 3D object from multiple 2D snapshots, to analyzing data from multiple points of view.</p><p><img src="https://rahatzamancse.netlify.app/images/post/mpse-mds/contribution.png"></p>
<p><div class="notices info" data-title="Info">
The following images are interactive!!!
</div><br />
<iframe src="https://rahatzamancse.netlify.app/others/post/mpse-mds/chair.html" style="border:none;height:500px;width:100%;" title="Chair"></iframe><br />
<iframe src="https://rahatzamancse.netlify.app/others/post/mpse-mds/desk.html" style="border:none;height:500px;width:100%;" title="Desk"></iframe></p>
<div class="notices info" data-title="Info">
All of the experiments done for this research are managed by neptune.ai which can be found in <a href="https://app.neptune.ai/rahatzamancse/MPSE-with-GAN/experiments?split=tbl&dash=charts&viewId=52255a62-06ba-4223-8940-c2d840d58d1e">this link</a>.
</div>
<h2 id="multi-perspective-simultaneous-embedding-with-mdshttpsarxivorgabs190906485"><a href="https://arxiv.org/abs/1909.06485">Multi-Perspective Simultaneous Embedding with MDS</a></h2>
<div class="notices info" data-title="Info">
A single page quick review of this paper can be found <a href="https://drive.google.com/file/d/1-dtJw2hgenwCvndu85WZotSLk4MvSD_d/view?usp=sharing">here</a>.
</div>
<p>MPSE stands for Multi-Perspective Simultaneous Embedding which is a method proposed by <a href="https://www2.cs.arizona.edu/~kobourov/">Dr. Stephen Kobourov</a> et. al. for visualizing high-dimensional data, based on multiple pairwise distances between the data points. Specifically, MPSE computes positions for the points in 3D and provides different views into the data by means of 2D projections (planes) that preserve each of the given distance matrices. There are two versions of it: fixed projections and variable projections. MPSE with fixed projections takes as input a set of pairwise distance matrices defined on the data points, along with the same number of projections and embeds the points in 3D so that the pairwise distances are preserved in the given projections. MPSE with variable projections takes as input a set of pairwise distance matrices and embeds the points in 3D while also computing the appropriate projections that preserve the pairwise distances. The proposed approach can be useful in multiple scenarios: from creating simultaneous embedding of multiple graphs on the same set of vertices, to reconstructing a 3D object from multiple 2D snapshots, to analyzing data from multiple points of view.</p>
<p>Using this point embedding algorithm, we have decided to use it for 3D model reconstruction from multiple 2D images. So we propose a novel algorithm to embed image keypoints in a 3D image. The problem assumes a set of image keypoints in an unknown 3D image and several snapshots (2D pictures) of this scene containing these keypoints, either partially of possibly all of the points. Given the coordinates of these keypoints in each snapshot the problem asks to recover their positions in the original 3D image such that the corresponding projections.</p>
<p>There are a lot of 3D reconstruction methods, especially in the field of computer vision with deep learning. But our proposal stands novel and benificial over many ways than deep learning which keep it’s use-case separate from those of deep learning applications. We do not need any supervised training or any pre-existing dataset. As a result, we also do not have the risks that come with deep learning, like overfitting, high training time or data acquisition phase complexity.</p>
<h2 id="3d-reconstruction-with-mpse">3D Reconstruction with MPSE</h2>
<p>3D reconstruction using MPSE would have been a straight forward application problem if the perspective images were ideal, e.g. all the points of a single object can be visible in all the images. But it is impossible for all the sides or points of an object to be visible from all the sides. So we had to come up with modifications to the original MPSE algorithm to come up with hidden points (points that are visible from at least 1 viewpoint but not all viewpoints).</p>
<p>Assume we have in total $N$ keypoints that we aim to embed in 3D. However, each of the given view for $1 \le k \le K$ does not include all of these $N$ points. We assume we are given parameters $\alpha_i^k$ for $1 \le k \le K$ and $1 \le i \le N$ which is 1 if the $i$-th points is visible in $k$-th view and 0 otherwise. we can alternatively introduce variables $\alpha_{ij}^k$ which would have a value $1$ if both $i$-th and $j$-th keypoints are visible in $k$-th view and 0 otherwise. The modified objective function for MPSE would be as follows<br />
% The objective function for MPSE with possible hidden points:<br />
\begin{equation}<br />
\sum_{k = 1}^K \sum_{i > j} \alpha^k_{ij} \left( D^k_{i j} - \Vert P^k(x_i) - P^k(x_j) \Vert \right)^2.<br />
\end{equation}<br />
Where $\alpha^k_{ij}$ is either $1$ or $0$. It is 1 when both points $i$ and $j$ are present in projection $k$ otherwise it is 0.<br />
Similarly, if with the use of $\alpha_i^k$, the objective dunctiuon would have the following form:<br />
\begin{equation}<br />
\label{eq:MPSE_hidden_points}<br />
\sum_{k = 1}^K \sum_{i > j} \alpha^k_{i} \alpha^k_{j} \left( D^k_{i j} - \Vert P^k(x_i) - P^k(x_j) \Vert \right)^2.<br />
\end{equation}<br />
where $\alpha^k_{i} = 1$ if the $i$-th point is present and set $\alpha^k_{i} = 0$ if the point is missing.</p>
<p>Our overall contribution in this is shown in the image at the top of this page. Some of the 3D pointsclouds that were reconstructed from only 3 images taken from different positions is given below:</p>
<h3 id="dataset-requirements">Dataset Requirements</h3>
<p>The exact dataset that MPSE needs should have the following features:</p>
<ol>
<li>A set of perspectives of a pointcloud. A perspective is a set of points of a specific pointcloud transformed or projected onto n-dimension. The goal is to reconstruct the pointcloud in m-dimension where $n < m$.</li>
<li>An annotation of correspondence of each point in different perspectives.</li>
</ol>
<h3 id="evaluation">Evaluation</h3>
<p>The evaluation of the MPSE algorithm is done based on different parameters of the model and the dataset.</p>
<p>The dissimilarity of two pointcloud is calculated using equation the below equation. Here $x_i$ is the $i$th point of newly generated point cloud and $y_i$ is $i$th point of the ground truth pointcloud.</p>
<p>$$\mathcal{L}(x, y) = \frac{\sum{||x_i - y_i||_2}}{n}$$</p>
<p>The loss function which is optimize For alignment of two pointclouds before calculating the distance is given</p>
<p>\begin{equation}<br />
min_{T}\mathcal{L}(T.x, y)<br />
\end{equation}</p>
<p>Here, $T$ is a 4*4 function and $x$ is the vector of generated point cloud in homogeneous coordinates. $y$ is also of the same form.</p>
<p>After having trails of different optimization functions, we found Adam optimizer to converge the fastest. The whole points alignment algorithm is expressed in the following figure:</p>
<p><img src="https://rahatzamancse.netlify.app/images/post/mpse-mds/alignment.png"></p>
<h4 id="performance-on-different-number-of-points">Performance on different number of points</h4>
<p><img src="https://rahatzamancse.netlify.app/images/post/mpse-mds/points-rmse.png"></p>
<h4 id="performance-on-different-number-of-hidden-points">Performance on different number of hidden points</h4>
<p><img src="https://rahatzamancse.netlify.app/images/post/mpse-mds/visiblepoints-rmse.png"></p>
<h4 id="performance-on-different-number-of-perspective-views">Performance on different number of perspective views</h4>
<p><img src="https://rahatzamancse.netlify.app/images/post/mpse-mds/perspective-time.png"><br />
<img src="https://rahatzamancse.netlify.app/images/post/mpse-mds/viewpoints-rmse.png"></p>
<div class="notices info" data-title="Info">
This work will be submitted to <a href="https://britishmachinevisionassociation.github.io/bmvc">BMVC</a>
</div>Rahat Zamanfeatured imagemachine-learningvisualizationresearchmpse
- 3D Reconstruction with GANhttps://rahatzamancse.netlify.app/posts/mpse-gan/Tue, 16 Feb 2021 13:22:57 -0700rahatzamancse@gmail.com (Rahat Zaman)Fri, 17 Jun 2022 16:59:44 -0700https://rahatzamancse.netlify.app/posts/mpse-gan/<p><img src="https://rahatzamancse.netlify.app/images/post/mpse-mds/contribution.png"></p>
<p><div class="notices info" data-title="Info">
The following images are interactive!!!<br />
Rotate each of the 3D pointclouds horizontally 90° to see the other view.
</div><br />
<iframe src="https://rahatzamancse.netlify.app/others/post/mpse-gan/square-circle.html" style="border:none;height:500px;width:100%;" title="Square and Circle"></iframe><br />
<iframe src="https://rahatzamancse.netlify.app/others/post/mpse-gan/hello-bye.html" style="border:none;height:500px;width:100%;" title="HELLO and BYE"></iframe></p>
<p>The next one is a little complicated with real binary images. The reconstruction is not great, but we can see similarity in the image and perspective easily.</p>
<p><img src="https://rahatzamancse.netlify.app/images/post/mpse-gan/tree.bmp" width="50%" style="float: left;"><img src="https://rahatzamancse.netlify.app/images/post/mpse-gan/lenna.png" width="50%"></p>
<iframe src="https://rahatzamancse.netlify.app/others/post/mpse-gan/lenna-tree.html" style="border:none;height:500px;width:100%;" title="Lenna and Tree"></iframe>
<div class="notices info" data-title="Info">
All of the experiments done for this research are managed by neptune.ai which can be found in <a href="https://app.neptune.ai/rahatzamancse/MPSE-mds/experiments?split=tbl&dash=leaderboard&viewId=72532e6d-1746-460a-ad85-ac994ec8cfb9">this link</a>.
</div>
<h4 id="disclaimer">Disclaimer</h4>
<p>This work is done as a <em>side project</em> of <a href="https://rahatzamancse.netlify.app/posts/mpse-mds/">MPSE MDS</a>. Without any duplications, I would suggest to go through that post first before this one.</p><p><img src="https://rahatzamancse.netlify.app/images/post/mpse-mds/contribution.png"></p>
<p><div class="notices info" data-title="Info">
The following images are interactive!!!<br />
Rotate each of the 3D pointclouds horizontally 90° to see the other view.
</div><br />
<iframe src="https://rahatzamancse.netlify.app/others/post/mpse-gan/square-circle.html" style="border:none;height:500px;width:100%;" title="Square and Circle"></iframe><br />
<iframe src="https://rahatzamancse.netlify.app/others/post/mpse-gan/hello-bye.html" style="border:none;height:500px;width:100%;" title="HELLO and BYE"></iframe></p>
<p>The next one is a little complicated with real binary images. The reconstruction is not great, but we can see similarity in the image and perspective easily.</p>
<p><img src="https://rahatzamancse.netlify.app/images/post/mpse-gan/tree.bmp" width="50%" style="float: left;"><img src="https://rahatzamancse.netlify.app/images/post/mpse-gan/lenna.png" width="50%"></p>
<iframe src="https://rahatzamancse.netlify.app/others/post/mpse-gan/lenna-tree.html" style="border:none;height:500px;width:100%;" title="Lenna and Tree"></iframe>
<div class="notices info" data-title="Info">
All of the experiments done for this research are managed by neptune.ai which can be found in <a href="https://app.neptune.ai/rahatzamancse/MPSE-mds/experiments?split=tbl&dash=leaderboard&viewId=72532e6d-1746-460a-ad85-ac994ec8cfb9">this link</a>.
</div>
<h4 id="disclaimer">Disclaimer</h4>
<p>This work is done as a <em>side project</em> of <a href="https://rahatzamancse.netlify.app/posts/mpse-mds/">MPSE MDS</a>. Without any duplications, I would suggest to go through that post first before this one.</p>
<div class="notices success" data-title="Success">
<h1 id="presentations">Presentations</h1>
<ul>
<li><a href="https://drive.google.com/file/d/1KRmVA_eVv3f7uYhcweAJ_4bfczukkDGS">Presentation 1</a></li>
<li><a href="https://drive.google.com/file/d/1s_K0CTFgkqMeYbOFoEh9CIUg1t-wdTRu/view?usp=sharing">Presentation 2</a></li>
<li><a href="https://drive.google.com/file/d/1rD8vjERyHiwLNa4QSIAVPmSvogd2zGMR/view?usp=sharing">Presentation 3</a></li>
</ul>
</div>
<h1 id="introduction">Introduction</h1>
<p>Following the original MPSE paper, I tried to implement the algorithm by myself. But, initially I could not get any result with my own implementation, maybe then I did not understood how SGD works with the given stress function. It seems that Stochastic Gradient Descent was not able to minimize the stress function at all (in my implementation).</p>
<p>\begin{equation}<br />
\underset{x \in !R ^{p \times n} }{\text{minimize}} \sum\limits_{k=1}^3 \sum\limits_{i > j} (D_{ij}^k - || P^k(x_i) - P^k(x_j) ||)^2 \<br />
\label{eq:stressmin}<br />
\end{equation}</p>
<p>Later, I started to think about how this can be done in many other ways. I found a new approach which I named mpse-gan.</p>
<h1 id="mpse-gan">MPSE-GAN</h1>
<p>Primarily, what 3D reconstruction with mpse is doing is generating a set of 3D points that look like a given set of points from the real world from a 2D perspective view. This is exactly what Generative Adversarial Networks can be very good at.</p>
<p>We tried a discriminating neural network with a slightly modified loss function of yours to predict whether a 3D representation presents all the 2D images from some perspectives. The motivation of this came from the Kaggle contest “3D MNIST” where the popular MNIST dataset is given as voxel (volumetric pixels) image and the task is to classify those 3D images. As seen in many example kernels in the contest, deep learning seems to be outperforming most of other optimization models. So, applying deep learning in this context yields high expectations.</p>
<p>Although I did not had enough experience working with 3D images at that time, I have studied about 3D convolution, Computer Vision, Geometric analysis and how to implement those. I have prior experience with GAN which helped me in this research. A theoretical architecture is given in the below figure and a data generation pipeline (IVBA) is tested and proposed on the next figure.</p>
<p><img alt="structure" src="https://rahatzamancse.netlify.app/images/post/mpse-gan/gan.png"><br />
<img alt="IVBA" src="https://rahatzamancse.netlify.app/images/post/mpse-gan/aug.png"></p>
<p>Some examples of the actual implementation is shown at the top of this page.</p>
<p>The actual mathematical explanation can be shown in terms of the original equations with GAN, but with multiple discriminator networks.</p>
<p>\begin{equation}<br />
D = Discriminator<br />
\end{equation}<br />
\begin{equation}<br />
G = Generator<br />
\end{equation}<br />
\begin{equation}<br />
\theta_d = Parameters of discriminator<br />
\end{equation}<br />
\begin{equation}<br />
\theta_g = Parameters of generator<br />
\end{equation}<br />
\begin{equation}<br />
P_z(z) = Input noise distribution<br />
\end{equation}<br />
\begin{equation}<br />
P_{data}(x) = Original data distribution<br />
\end{equation}<br />
\begin{equation}<br />
P_g(x) = Generated distribution<br />
\end{equation}</p>
<p>The Cross-Entropy loss function for the reconstruction process is:</p>
<p>\begin{equation}<br />
\begin{split}<br />
Loss(\hat{y}, y) = y \times log \hat{y} + (1-y) \times log(1-\hat{y})<br />
\end{split}<br />
\end{equation}</p>
<p>So for the whole dataset, it will be:</p>
<p>\begin{equation}<br />
Loss(D(x), 1) = log(D(x)) \<br />
\end{equation}<br />
\begin{equation}<br />
Loss(D(G(z)), 0) = log(1-D(G(z)))<br />
\end{equation}</p>
<p>Objective function of Discriminator networks:</p>
<p>\begin{equation}<br />
\begin{split}<br />
Loss(D)_{min} \longrightarrow max[log(D(x)) + log(1-D(G(z)))]<br />
\end{split}<br />
\end{equation}</p>
<p>Objective function of Generative networks:</p>
<p>\begin{equation}<br />
Loss(G)_{min} \longrightarrow min[log(D(x)) + log(1-D(G(z)))]<br />
\end{equation}</p>
<p>So Overall Objective function will be:</p>
<p>\begin{equation}<br />
min_G max_D [log(D(x)) + log(1 - D(G(z)))]<br />
\end{equation}</p>
<h2 id="iterations">Iterations</h2>
<p>At each of the iterations, we can see how the reconstruction is actually happening at each iteration.</p>
<p><img src="https://rahatzamancse.netlify.app/images/post/mpse-gan/it-hello.png" width="50%" style="float: left;"><img src="https://rahatzamancse.netlify.app/images/post/mpse-gan/it-bye.png" width="50%"></p>
<p><img src="https://rahatzamancse.netlify.app/images/post/mpse-gan/it-circle.png" width="50%" style="float: left;"><img src="https://rahatzamancse.netlify.app/images/post/mpse-gan/it-square.png" width="50%"></p>
<div class="notices info" data-title="Info">
You can click on each image to zoom in.
</div>
<p>All the corresponding loss values at each iteration can be found in the <a href="https://app.neptune.ai/rahatzamancse/MPSE-with-GAN/e/MPSEG-86/charts">neptune experiment website</a>.</p>
<p><br/><hr/></p>
<div class="notices info" data-title="Info">
This work will be submitted to <a href="https://britishmachinevisionassociation.github.io/bmvc">BMVC</a>
</div>Rahat Zamanfeatured imagemachine-learningvisualizationGANresearchmpse