Szabo, Z;
Lőrincz, A;
(2009)
Fast Parallel Estimation of High Dimensional Information Theoretical Quantities with Low Dimensional Random Projection Ensembles.
In: Adali, T and Jutten, C and Romano, JMT and Barros, AK, (eds.)
Independent Component Analysis and Signal Separation: 8th International Conference, ICA 2009, Paraty, Brazil, March 15-18, 2009. Proceedings.
(146 - 153).
Springer-Verlag Berlin Heidelberg
![[thumbnail of szabo09fast.pdf]](https://discovery.ucl.ac.uk/style/images/fileicons/application_pdf.png) Preview |
PDF
szabo09fast.pdf Available under License : See the attached licence file. Download (329kB) |
Preview |
PDF
szabo09fast_poster.pdf Available under License : See the attached licence file. Download (303kB) |
Abstract
The estimation of relevant information theoretical quantities, such as entropy, mutual information, and various divergences is computationally expensive in high dimensions. However, for this task, one may apply pairwise Euclidean distances of sample points, which suits random projection (RP) based low dimensional embeddings. The Johnson-Lindenstrauss (JL) lemma gives theoretical bound on the dimension of the low dimensional embedding. We adapt the RP technique for the estimation of information theoretical quantities. Intriguingly, we find that embeddings into extremely small dimensions, far below the bounds of the JL lemma, provide satisfactory estimates for the original task.We illustrate this in the Independent Subspace Analysis (ISA) task; we combine RP dimension reduction with a simple ensemble method. We gain considerable speed-up with the potential of real-time parallel estimation of high dimensional information theoretical quantities.
| Type: | Book chapter |
|---|---|
| Title: | Fast Parallel Estimation of High Dimensional Information Theoretical Quantities with Low Dimensional Random Projection Ensembles |
| Event: | International Conference on Independent Component Analysis and Signal Separation (ICA) |
| Location: | Paraty, Brazil |
| Dates: | 2009-03-15 - 2009-03-18 |
| ISBN-13: | 978-3-642-00598-5 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1007/978-3-642-00599-2_19 |
| Publisher version: | http://dx.doi.org/10.1007/978-3-642-00599-2_19 |
| Language: | English |
| Additional information: | The final publication is available at Springer via http://dx.doi.org/10.1007/978-3-642-00599-2_19 |
| Keywords: | independent subspace analysis, information theoretical estimations, pairwise distances, random projection |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences |
| URI: | https://discovery.ucl.ac.uk/id/eprint/1433164 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
