Advances in Independent Component Analysis and Learning Machines

In honour of Professor Erkki Oja, one of the pioneers of Independent Component Analysis (ICA), this book reviews key advances in the theory and application of ICA, as well as its influence on signal processing, pattern recognition, machine learning, and data mining.

Examples of topics which have developed from the advances of ICA, which are covered in the book are:

• A unifying probabilistic model for PCA and ICA
• Optimization methods for matrix decompositions
• Insights into the FastICA algorithm
• Unsupervised deep learning
• Machine vision and image retrieval

• A review of developments in the theory and applications of independent component analysis, and its influence in important areas such as statistical signal processing, pattern recognition and deep learning
• A diverse set of application fields, ranging from machine vision to science policy data
• Contributions from leading researchers in the field

• Language: English
• Publisher: Academic Press
• Release date: May 14, 2015
• ISBN: 9780128028070

Book preview

China

Introduction

Ricardo Vigário

As this book celebrates some great scientific avenues inspired by visionary researchers of the likes of Erkki Oja, I believe it is of particular relevance to remember the person, his career, and the shoulders from where several such amazing prospects sprung forth.

Many are the topics of research where Erkki has brought a marked contribution. Many more are the researchers, young and older, who shared and shaped paths of their working careers with him. For the sake of conciseness and a certain focus, I took the liberty to invite only a few of Erkki’s distinguished friends and colleagues to contribute with some words describing their relation to him. We will see in all of them the same admiration and recognition for a true scholar, but also some echoes of the person and the communicator, who often transcends the limits of the work rules, and ventures into the friendship realm.

This chapter is admittedly the nonscientific contribution in the book. Therefore, instead of thoroughly editing its text, in search for a rather cohesive structure, I will opt for a more platonic approach: reality will always exceed all finite projections we may find for it. In the cave wall that is this book, I will therefore let each of the participants voice their own views about Erkki. Those projections will certainly not be uncorrelated, let alone independent, as they address and characterize a common multi-dimensional source. Yet, the subspace spanned by the following words will hopefully draw an impressionist sketch of the person, the scholar, and the friend.

Using the liberty that assists the convener of such a delightful forum, I will take the opportunity to open with my own, short contribution.

Erkki Oja and his learning rule. Original photo by Anni Hanén.

A Student and a Co-Worker

In the early 1990s, I contacted a professor in Portugal on the subject of finding a suitable supervisor for a postgraduate research work. I thought I knew exactly what I wanted to do, but was rather uncertain where to go, and whom to work with. That professor, also in this chapter (LBA), suggested two names he knew who were leading researchers in the area I wanted to study – unsupervised learning, with biologically plausible computer vision goals. One worked in a remote city in Finland, far away from its capital. The other, Ralph Linsker, had a research group at IBM Thomas J. Watson Res. Center, in New York, USA. I thought: maybe I can spend this year in Finland, and then move to IBM for the doctoral degree. This decision was made even easier because Erkki Oja had just moved to Otaniemi, a campus in the thriving capital region.

Over 20 years later, after experiencing research at several other leading machine learning groups, I find myself still working in the same department, idealized by academician Teuvo Kohonen, and developed by Erkki Oja to very high international standards. One could wrongfully mistake this persistence for inertia. None could be farther from the truth. It was very clear, from my very first encounter with Erkki and his research group, that his department was a perfect place to germinate new ideas, and even propose new research directions. At the time, it had a recognized group of experts in Neural Networks, carrying out leading research in unsupervised learning. Equally important in my decision was the fact that the university campus comprised, among many other excellent research areas, an internationally recognized brain research unit, led by the expert hands of researchers, such as the academicians Olli V. Lounasmaa and Riitta Hari.

One cannot say that neuroinformatics was then at the core of Erkki’s research interests. Yet, he had always a finger on the pulse of science, and saw that a bridge between the aforementioned areas of research excellence was a valuable asset in the development of his own research endeavors. As one example, under his mentoring, pioneering research on biomedical applications of independent component analysis (ICA) was proposed. This is still, to date, one of the most accomplished areas of applied research for ICA.

The two anecdotal stories above reflect well Erkki’s nature: an acute scholar, with an amiable nature; a visionary in science, as well as a true mentor, empowering his junior colleagues, leading and supporting them to take independent responsibility in science; and with a door always open, and a word of guidance at all times.

With permission, I will end these lines with the words of academician Riitta Hari: over the years, I have had the privilege of interacting with Erkki at a scientific, academic, and personal level, and the interaction has always been smooth and effective. I highly appreciate Erkki as a scientist and a colleague. I am sure that his outstanding research and pedagogical mentoring will continue to promote science in multiple disciplines.

Prof. Simon Haykin

A novel property of Neural Networks and Learning Machines is their inherent ability to learn from the environment; and through learning, improve their performance in some statistical sense. Work in such research fields, right from their inception, has been motivated by the recognition that the human brain is a powerful information processing machine, which distinguishes itself, in a remarkable manner, from the digital computer.

Professor Erkki Oja’s influential scientific work spans over four highly prolific decades, from early research on associative memories in the late 1970s and early 1980s. To elaborate, Hebbian learning and principles of subspace analysis are basic to pattern recognition and machine vision, as well as blind source separation (BSS) and ICA, fields in which Prof. Oja researched throughout the 1990s and early 2000s. More recently, nonnegative matrix factorization and computational inference came into prominence. Throughout all that time, the points made herein apply to an insatiable thirst for knowledge, and an exceptional ability to detect and discover new trends in Neural Computation. Simply put, all of the above are the hallmark of a distinguished academic, namely, Prof. Erkki Oja.

A highly remarkable learning rule, known as Oja’s rule, so-called in recognition of the work done by Prof. Oja, was published in 1982. The rule was motivated by Hebb’s postulate of learning, which was first described in a book written by the neuropsychologist Donald Hebb in 1949. For the record, Oja’s rule may be described as follows:

A single linear neuron with a Hebbian-type adaptation rule for its synaptic weights can evolve into a filter for the first principal component of the input distribution.

The rule is simple to state, yet it is very rich in its mathematical exposé.

Furthermore, Prof. Oja and his research teams, over the years, went on to expand his learning rule for the identification of eigensubspaces, nonlinear principal component decompositions, and ICA algorithms. In addition to very elegant and efficient theoretical advances in Neural Computation and related topics, Prof. Oja always sought their use in practice, supporting pioneering research in many ambitious application areas: computer vision and pattern recognition; neuroinformatics and biomedical engineering applications; as well as proactive information retrieval and inference.

To conclude, Prof. Oja is an innovator par excellence. Knowing him as I do, he will continue to impact the world of Neural Computation through his pioneering contributions in years to come.

Prof. José Príncipe

In the 1970s, Erkki Oja opened up roads to many discoveries in the late portion of the twentieth century and beyond. His first works on PCA provided the tone which blended a very solid grounding in mathematics with the amazing power of online adaptation that we still are grasping to fully understand. They require imagination, intuition, and transcend the aseptic world of mathematics. Erkki predicted and solidified many important applications of his simple adaptation rule (now with his name), and the applications of FastICA to imaging will always be connoted with him and his group in Finland.

His leadership in adaptive informatics, and more recently in computational inference, has propelled many young scientists in Finland, and all around the world to this exciting domain so important for big data. But on top of it all, Erkki’s legacy transcends science and engineering. He is a true scholar, a gentleman, a wonderful person, and I am very fortunate to call him my friend.

Prof. Tülay Adali

Erkki Oja has been a true leader in the field, and paved the way to much of the exciting work going on today in the machine learning field, particularly in nonlinear adaptive processing. He has provided the essential bridge between neural computation and adaptive optimization theory, and has not only provided the tools to address many of today’s challenging problems but also offered different and fruitful ways to visualize them, making his impact particularly long lasting. The continuing strong rate in citations to his work from all periods, including those from the early 1980s, is a simple testament to the influence of his work and its continuing importance.

Beyond all this, he does inspire those around him, not only to achieve their best technically but also to enjoy life. It has been always a pleasure to be in his company, which might include sharing a fine meal with a nice glass of wine, or following the rhythm of the music, as he demonstrates to those around him how to truly be in the moment and enjoy life.

Prof. Luís Borges de Almeida

I first met Erkki Oja in 1990 at a workshop on Neural Networks that I helped to organize in Sesimbra, Portugal. Over the years, I have had the opportunity to collaborate with him in a number of circumstances, the major one having been participation in the BLISS research project, which extended from 2000 to 2003.

I have come to deeply appreciate his scientific competence and good judgment, as well as his good disposition and sense of humor. He has made important scientific contributions to several fields, especially Neural Networks, ICA, and BSS. He has attracted many people of great quality to work with him, having given a great contribution to their scientific training, and having formed a renowned scientific lab.

Among all these facts that I truly value, there is one that I value the most: Erkki Oja has become a very good friend, beyond all the scientific and professional cooperation. I wish to express here my best wishes for his future.

Prof. Christian Jutten

My Dear Erkki,

In a few lines, I would like to explain how you have been and you are an important person for me, at the scientific as well as human levels.

After my PhD in 1981, devoted to investigating how information is transmitted and modified through actual and formal neural networks, I considered studying learning and I have been strongly attracted by unsupervised learning, perhaps as a reaction against the trend of supervised learning.

Especially, I was very interested by Hebb’s rule and its extension, Oja’s rule. I was actually impassioned by the ideas, principles, and algorithms of Self-Organizing Maps (SOM) pioneered by T. Kohonen, and for which Erkki Oja did many important contributions.

At that time (and still now), I was wondering how living systems could be able to do so powerful processings, so difficult for a computer, and my researches were inspired by studying SOM, PCA, and other of your contributions in pattern recognition. I believe that probably the first algorithm of BSS, designed in 1983 with B. Ans and J. Hérault, for modeling how vertebrate can control joint motion, has been the fruit of the core question of unsupervised learning: how and what is it possible to learn without any supervisor?

I believe that I met you for the first time in 1986 in Snowbird…. Probably, you do not remember since I was a very young researcher. Later, I had a stronger contact with you in the 1990s, when our two groups submitted a proposal for a European project: unfortunately, it was not accepted. However, during the meetings for preparing the project, I have had the chance to meet you and discuss with you longer. Certainly, we became closer with the development of BSS, and the similarity between your nonlinear PCA and our first blind separation algorithms. Our two groups in Helsinki and in Grenoble contributed to design some building blocks of source separation methods, with various applications. Great moments were the first and the second ICA conference, in Aussois and then in Helsinski, and the European project BLISS in which we were partners.

In addition to the scientific respect I have for you, Erkki, I believe it was the beginning of a strong friendship. Finally, in 2008 and later in 2013, I applied for a very selective position in Institut Universitaire de France, and I asked you if you would like to act as one of my reference scientists: I am really honored that you accepted with pleasure, and each time you do the job with a perfect efficacy! Thank you a lot.

Erkki, you are now a distinguished Professor of Aalto University and you deserve it. Since 1981, you and your work have guided me and I am sure that you have been such a beacon for many other scientists, worldwide. You are my friend, and you are welcome in my lab and in my home. I wish you the best.

Prof. Mark Plumbley

My own research has been influenced by the pioneering work of Erkki Oja almost ever since I began my research career. Like many others who began their PhD research in artificial neural networks, or connectionist models, in the late 1980s, I had to decide between supervised and unsupervised learning as a research topic. Having worked a little on information theory and coding in video, I wondered if this could be used as a driver for unsupervised learning, for a suitable algorithm.

Like many others, I had come across the famous unsupervised Kohonen network, or SOM, in lectures in a Masters course, and I read up more about this in a text book by Kohonen. The SOM was perhaps too complex for what I was trying to do, but buried in the middle of Kohonen’s text book was an apparently simple Hebb-like learning algorithm for a linear neuron, which learned to find the normalized principal component of its input data set.

This was the Oja rule, of Erkki Oja’s classic 1982 paper. That rule was simple, beautiful, and intriguing. I had found the starting point for my PhD.

My PhD work investigated the generalization to multiple outputs, building on the work by Oja and Karhunen in 1985, which found either several ordered principal components, or the space spanned by the principal components (the principal subspace). I found ways to use Shannon information as a motivation for the algorithm (related to Ralph Linsker’s Infomax principle), and also to use information as an energy measure (Lyapunov function) to prove convergence of these Oja-like algorithms. Convergence analysis by others also demonstrated that the deceptively simple algorithm performed two complementary operations. For the original single neuron version, the weight vector finds the direction of the principal component, while simultaneously the length of the weight vector converges to unity. For the multiple-neuron version, the space spanned by the weight vectors converged to the principal subspace, while simultaneously the weight vectors themselves converged to an orthonormal set.

This magic self-regulation took a while to be understood properly by researchers. It meant, for example, that you cannot simply change the signs to obtain a minor-component analysis algorithm. While you would find the direction of the minor component, the self-regulation part would now be unstable and the length of the weight vector would diverge. So this apparently simple algorithm has given many researchers a lot of material to work on! In this world of Big Data, Oja’s algorithm is still finding interesting applications. In the recent Machine Learning for Signal Processing conference (MLSP 2014), it was used to explore PCA estimation in distributed networks of processors.

After my PhD, as a new lecturer at King’s College London, I helped establish an EU-funded Network of Excellence in neural networks, NEuroNet, where Erkki Oja was one of the key partners. As part of this I participated in a small specialist workshop on Independent and Principal Component Analysis Methods in Thessaloniki in 1996, hosted by Kostas Diamantaras. At the time I was still concentrated on linear PCA-related methods, but Erkki was by then already a pioneer in the emerging field of ICA, using nonlinear neural networks.

By 2000 I was reviewing my research direction, and finally realized that ICA was going to be an interesting area to work in. My first proper ICA conference was ICA 2001, hosted by Erkki on a small island near to Helsinki University of Technology. I knew I had a lot of catching up to do in the field of ICA, and Erkki agreed to host me for a 3-month visit in early 2002, funded by Leverhulme Trust. This visit gave me the chance to work closely with Erkki and his lab, and was instrumental in refocusing my research into ICA and source separation. It gave me the chance to work with Erkki on nonnegative versions of his nonlinear PCA model: this became the nonnegative PCA algorithm for nonnegative ICA (NN-ICA). It also gave me the chance to see how well Erkki is regarded as a research leader by colleagues, collaborators, and other staff in his lab, as well as his support and kindness to me. I still have the pair of Moomin mugs that he gave me as a souvenir of my visit!

Erkki has continued to be an inspiration to my research career. With the support of Erkki and others I hosted the International Conference on Independent Component Analysis and Signal Separation (ICA 2007) in London a few years later, and chaired the ICA Steering Committee. The work with Erkki on nonnegative ICA also led to an interest in geometrical methods of source separations, sparse representations, and compressed sensing, which still drives my current research. I wish Erkki all the best for his forthcoming retirement, and I am sure that he will continue to inspire me and many other researchers for many years to come.

Prof. Klaus-Robert Müller and Dr. Andreas Ziehe

Erkki, whom one of us (KRM) already met in 1991, at the occasion of the first ICANN Conference, in Helsinki, was already then an idol and my hero in neural networks. Definitely one of the great pioneers of our field. I respectfully called him Prof. Oja then, as a PhD student with still more than 1 year to go, and experienced his wonderful kindness, open mind, and his friendly advice. ICANN’91 – my first international conference where I gave a talk – certainly was pivotal in my career, since I met many other researchers there for the first time, for example, Geoff Hinton, Shun-Ichi Amari, Teuvo Kohonen, John Hertz, Werner von Seelen, and also the then younger folks Klaus Obermayer, Helge Ritter, Thomas Martinetz, Klaus Pawelzik, among