An ICA page-papers,code,demo,links (Tony Bell)

An ICA page - papers, code, demos, links

Explanation: ICA is about factoring probability distributions,and doing blind source separation. It is related to lots of other things- entropy and information maximisation, maximum likelihood densityestimation(MLE), EM (expectation maximisation, which is MLE with hiddenvariables)andprojection pursuit. It is basically a way of finding special linear(non-orthogonal)co-ordinate systems in multivariate data, using higher-order statisticsin various ways. If you don't understand, read the papers below!

Applications: anywhere you have ensembles of multivariatedata,eg: anywhere you might use PCA (Principal Components Analysis). Examplesinclude blind separation (eg: of mixed speech signals), biomedical dataprocessing (eg: of EEG [brainwave] data), finding `features' in data(eg:learning edge-detectors for ensembles of natural images).

Algorithms: There are many different algorithms, but oftentheyare not so different really. Read on.

-Tony Bell

Papers.

A selection of papers that I have access to right now. There is nopretenceto completeness, so sorry if you're omitted! Email me if you havesomethingyou'd like to include. Many other papers are collected at ParisSmaragdis'ssite listed below.

Quick guide to the papers:
Our main work is in B1 and B3: infomax/ICA for source separation andimagecoding respectively. The related natural gradient approach (which we nowuse) is in A1, a special case of the algorithm proposed in CU. Therelationsbetween these algorithms and maximum likelihood (C1, PH) are gonethroughin C2, MK, PP and O1 (so we'd better believe it!). The infomax originscan be traced in NP. A nice review from the Finnish school appears inKA.D1 is a book containing much material on ICA. Clever extensions andanalyses are appearing in G1, PN, T1, T2, and finally PP, where temporalcontext gives better separation. Perhaps the best work on sourceseparation/deconvolution is in LA (a pun). Our work on EEGs andERPs is in M1 and M2.

[NOTE: All papers are PostScript file. Compressed versions areX-compressed.They have ".ps.Z" on the end, and need UNIX `uncompress' to makethem ".ps" files.]

[A1] Amari S. Cichocki A. and Yang H.H. 1996. A new learning algorithm for blind signal separation, Advances in Neural Information Processing Systems 8, MIT press. Paper

[A2] Amari S-I. 1997. Natural Gradient works efficiently in learning. submitted to Neural Computation Paper

[B1] Bell A.J. and Sejnowski T.J. 1995. An information maximisation approach to blind separation and blind deconvolution, Neural Computation, 7, 6, 1129-1159 Abstract , Paper (0.9MB), Compressed (0.3MB) (38 pages). [3 short conference papers on the same material: ICASSP 95 (1.4MB, 4 pages) , NIPS 94 (0.2MB, 8 pages), and NOLTA95 . ]

[B2] Bell A.J. and Sejnowski T.J. 1996a. Learning the higher-order structure of a natural sound, Network: Computation in Neural Systems, 7 Paper

[B3] Bell A.J. and Sejnowski T.J. 1996. The `Independent Components' of natural scenes are edge filters, to appear in Vision Research, [Please note that this is a draft] Paper (1.5MB) , Compressed (0.4MB) (27 pages).[*Compressed tar-file of figures in case they don't print properly when you print the paper].

[B4] Bell A.J. and Sejnowski T.J. 1996. Edges are the `independent components'of natural scenes,
Advances in Neural Information Processing Systems 9, MIT press. Paper

[C1] Cardoso J-F. and Laheld B. 1996. Equivariant adaptive source separation, IEEE Trans. on Signal Proc., to appear Paper

[C2] Cardoso J-F, 1997. Infomax and maximum likelihood for blind separation, to appear in IEEE Signal Processing Letters, Paper

[D1] Deco G. and Obradovic D. 1996. An information-theoretic approach to neural computing, Springer-verlag Book

[G1] Girolami M. and Fyfe C. 1996. Negentropy and kurtosois as projection pursuit indices provide generalised ICA algorithms. NIPS '96 working paper Paper

[KA] Karhunen J. 1996. Neural approaches to independent component analysis and source separation, Proc 4th European Symposium on Artificial Neural Networks (ESANN '96). Paper

[LA] Lambert R.H. 1996. Multi-channel blind deconvolution: FIR matrix algebra and separation of multipath mixtures, Ph.D. Thesis, Elec. Eng., Univ. of Southern California

[LB] Lee T-W., Bell A.J. and Lambert R. 1997. Blind separation of delayed and convolved sources
in Proceedings of NIPS*9 (1996) Paper

[L2] Lee T-W., Bell A.J. and Orglmeister R. 1997. Blind Separation of Real -World Signals,
in Proceedings of International Conference on Neural Networks (Houston), ICNN '97 Paper

[L3] Lee T-W. Girolami M., Bell A.J. and Sejnowski T.J. 1998. A unifying Information-theoretic framework for Independent Component Analysis. International Journal on Mathematical and Computer Modeling, in press. Paper

[MK] MacKay D. 1996. Maximum Likelihood and covariant algorithms for Independent components analysis, DRAFT 3.1 Paper

[M1] Makeig S., Bell A.J., Jung T-P. and Sejnowski T.J. 1995. Independent Component Analysis of Electroencephalographic Data, in Mozer M. et al (eds) Advances in Neural Information Processing Systems 8, MIT press Paper

[M2] Makeig S., Jung T-P., Bell A.J., Ghahremani D. and Sejnowski T.J. 1997. Blind Separation of Event-related Brain Responses into Independent Components, Proc. Natl. Acad. Sci. USA
to appear Paper

[NP] Nadal J-P. and Parga N. 1994. Non-linear neurons in the low-noise limit: a factorial code maximises information transfer, Network, 4:295-312, Paper

[O1] Olshausen B. 1996. Learning linear, sparse, factorial codes, MIT AI-memo No. 1580 , aper

[PN] Parga N and Nadal J-P. 1996. Blind source separation with time-dependent mixtures, Signal Processing, submitted. Paper

[PP] Pearlmutter B.A. and Parra L.C. 1996. A context-sensitive generalization of ICA, Proc. ICONIP '96, Japan Paper

[PF] Platt J.C. and Faggin F. 1992. Networks for the separation of sources that are superimposed
and delayed, NIPS 1992 Paper

[T1] Torkkola K. 1996a. Blind separation of delayed sources based on information maximisation, Proc. ICASSP, Atlanta, May 1996 Paper

[T2] Torkkola K. 1996b. Blind separation of convolved sources based on information maximisation, Paper

The following two papers are probably under-cited, containing,respectively,`early' theoretical and practical insights into algorithms which arepartof the infomax/maximum-likelihood/natural-gradient family of ICAalgorithms.Unfortunately we don't have them online. Does anyone else? Also, let usnot forget the French originators: Herault/Jutten, Comon and others.

[PH] Pham D.T. Garrat P and Jutten C. 1992. Separation of a mixture of independent sources through a maximum likelihood approach, in Proc. EUSIPCO, p.771-774

[CU] Cichocki A., Unbehauen R., \& Rummert E. 1994. Robust learning algorithm for blind separation of signals, Electronics Letters, 30, 17, 1386-1387

Code.

Basic ICAcode in MATLAB (as used in Bell and Sejnowski 1996) It's simple.

ScottMakeiget al's MATLAB 4.2 routines for applying ICA to psychophysiologicaldata (tar-file)

Demos.

Demoofreal-room blind separation/deconvolution of two sources

(by Te-Won Lee . Uses infomax/ICA techniques in freq. domain with FIR matrix methods,
pioneered by Russ Lambert at USC. See our NIPS*96 poster - paper [LB] above.)

Links.

Hastily assembled selection of Web pages with more ICA papers anddetails.
Paris Smaragdis' page has a huge amount of material. It is more-or-lessa
superset of this page.

ParisSmaragdis'ICA & BSS page (MIT) GO TO THIS PAGE. IT'S GREAT.

AllanBarros'sICA page in Japan. Another huge collection of stuff.

Jean-François Cardoso

AndrzejCichocki

Erkki Oja

Scott Makeig,

Shun-ichiAmari

J.-P. Nadal :Publications

MarkGirolami's Home Page

Homepage of Aapo Hyvärinen

NIPS*96SignalProcessing Workshop (several new papers).

The WWW homepage of JuhaKarhunen

BSSResearch(Experiments at RIKEN in Japan)