UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Novel second-order techniques and global optimisation methods for supervised training of multi-layer perceptrons

Shepherd, Adrian John; (1995) Novel second-order techniques and global optimisation methods for supervised training of multi-layer perceptrons. Doctoral thesis (Ph.D), UCL (University College London). Green open access

[img] Text
Novel_second-order_techniques_.pdf

Download (5MB)

Abstract

Conventional training methods for multi-layer perceptrons (MLPs), derived from the traditional backpropagation algorithm, have three serious inadequacies; convergence to a solution is frequently slow; they do not always converge to the desired global solution; and their performance is highly dependent on the setting of one or more user-defined parameters. A growing body of research indicates that second-order training methods, derived from classical optimisation theory, offer substantial improvements in training speed and a reduced sensitivity to initial parameter settings. However, experiments conducted for this research suggest that most second-order methods have worse global convergence properties than conventional methods. On the other hand, training methods that are designed to have better global convergence characteristics than conventional methods - for example, stochastic training methods - are typically as slow or slower than conventional methods. The aim of this research is to develop MLP training algorithms that are both fast and 'globally-reliable' by combining second-order methods with a novel deterministic strategy for global optimisation. Expanded Range Approximation (ERA). Unlike most stochastic methods for global optimisation, the implementation of ERA with a second-order algorithm is trivial. When tested on benchmark training tasks, hybrid second-order/ERA algorithms (with appropriate parameter settings) were considerably faster and converged to a global minimum as or more frequently than conventional algorithms. This thesis also gives practical guidelines for the efficient implementation of second-order training algorithms, with particular attention paid to factors that affect the probability of a given algorithm attaining a global minimum. In addition, a novel line-search algorithm is presented that offers an efficient compromise between the reliability of safeguarded polynomial interpolation and the speed of backtracking line searches; used as part of a second-order training algorithm, only a single function evaluation is required per training iteration in the best case.

Type: Thesis (Doctoral)
Qualification: Ph.D
Title: Novel second-order techniques and global optimisation methods for supervised training of multi-layer perceptrons
Open access status: An open access version is available from UCL Discovery
Language: English
Additional information: Thesis digitised by ProQuest.
URI: https://discovery.ucl.ac.uk/id/eprint/10103818
Downloads since deposit
19Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item