Mar 23, 2009
Originally posted on Half an Hour, March 23, 2009.
The Network Phenomenon: Empiricism and the New Connectionism
Stephen Downes, 1990
(The whole document in MS-Word)
TNP Part X Previous Post
XI Projects and Investigations
A. Computational Difficulties
Let me now conclude this paper by outlining a number of areas of further investigation which ought to be pursued in order to accomplish the fullest and most useful presentation. These areas divide into two distinct categories: conceptual difficulties and computational difficulties. Let me outline some computational difficulties first.
By computational difficulties I mean aspects of the implementation of connectionist theories on computers. A number of concerns can be raised by viewing connectionism within a philosophical framework and some additional features are required. What I would like to do here is actually build a connectionist system using the C programming language and intended for application on an IBM XT compatible or clone. The large number of options, for example, different learning rules, will be incorporated as options on my own system. This system will fill a void on the market: an easy to use connectionist system which costs less than $1,000.
Having developed a connectionist system (which I'll call SDPDP) I want first to look at network variability in aconnectionist system. First of all, I want to construct nets in which different options may be employed in different parts of the same net at the same time. For example, in a PDP net [83] either every unit employs a stochastic on-off activation or every unit is activated in degrees. But in some systems, we want to be able to have units of both varities. In addition to variable structure, I want to incorporate some mechanisms of network plasticity. For in human systems, not only the connections, but the units themselves grow in response to input, especially in early life. Finally, I want to consider what I call "dimensions". For we want it to be the case that such things as religious conversions and scientific revolutions are possible. This requires that a network be able to construct [alternative] pairs of stable representations at the same time, which may alternate in priority. [84] Each of these alternative representations I cann a "dimension".
Another computational problem which I wish to consider concerns learning schedules and annealing. Currently, PDP systems employ a system which is very similar to that employed in physics. But, first, it is not clear that an annealing equation which is suitable for thermodynamics is suitable for human brains. I would like to investigate grounds for choosing one, rather than another, annealing equation. Second, it is clear to me that the annealing schedule employed is inadequate. In my view, temperature increases and decreases ought to be cyclic, as for example paterns of increased brain activity when we sleep. In addition, temperature ought to be sensitive to input, so that we can rapidly process conflicting input.
Finally, there is the hardware itself to think about. Human hardware is much smaller and more complex [than] contemporary computer technology. Perhaps we will not be able to build actual neurons, however, it seems reasonable that, now that we know exactly what we are looking for, we can make some plausible suggestions regarding how to build a computer neuron. I think that it would be best if many of the features currently represented by parameters, for example, threshold or rest values, can be implemented physically.
B. Conceptual Questions
By conceptual questions, I mean investigations into some of the things which connectionism can tell us about epistemology and the philosophy of mind. For, if the arguments concerning rules and categories are sufficiently strong, then we will want to reevaluate such concepts as knowledge and belief. For example, I would like to say that an item of knowledge is a stable pattern of activation, a pattern which tends not to change given varying input. If this is the case, then I may want to say with Feldman that "you do not have a store of knowledge, you are your knowledge." [85] In such a case, then, it becomes necessary to explore what we are and what part of us it is which is our knowledge.
In addition, I want to consider questions concerning theoretical and physical parallelism which arise. For example, through this paper I have used the terms "neuron" and "unit" roughly equivalently. I have also talked of the advisability of using this or that learning equation according to whether or not humans actually employ (or instantiate) the equation. We need to ask, first, whether or not we should design systems in parallel with human neural structure, and if so, what they would look like, and even further, how we would determine what they would look like.
As another investigation, I want to make some remarks about the nature of knowledge (as opposed to the definition of knowledge). For, if knowledge consists of stable patterns of activation then we cannot think in terms of knowledge as being sentences which have a given propositional content. It is unclear whether we can assign propositional content to patterns of activation. If that difficulty does arise, then we may want to consider some other relation between that which would serve as content (for example, representations of events in the real world) and patterns of activation. Here, perhaps, one could follow Armstrong and oldman and assert that there is a causal connection (and distinguish between appropriate and inappropriate causes). In order to successfuly defend this approach, it is necessary to give a fulla ccount of how we learn about causes.
Yet another investigation concerns consciousness. I have suggested above that there are conscious and unconscious regions of the brain. My belief is that those regions which are conscious are those which correspond to the activation of sensory input areas. In other words, my hearing someone speak a sentence and my thinking in a sentence is an activation of the same set of neurons (or an overlapping set). This solves the problem of how we can have a "stream of consciousness [86] in a non-linear network. But a much more detiled story is required here.
Finally, it is worth posing the question of whether connectionism is a type of scientific revolution, in the Kuhnian sense, or whether it is not. Some philosophers, for example Stich and Johnson-Laird, have expressed the opinion that it is not. In my own view, since so many traditional concepts must be overturned, then it is a scientific revolution. Haveing said that, however, I must ask whether or not we are working within an eliminativist paradign, as suggested by, say, Churcland, or not. In my view, there is still a role for words such as "knowledge" and "belief". If I believe this, then first I must explain this role, and then show how this role makes sense within the new paradigm.
C. Other Projects
When I began by asserting that connectionism vindicates empiricism, I embarked on a philosophical enterprise. What followed has been primarily technical and non-philosophical. I would like to return to a connectionist treatment of some philosophical issues.
For example, some contemporary [87] adocate a form of nominalism. While the philosophical debates concerning realism and nominalism are periphrial to this project, it is still the case that connectionism, if successful, should shed some light in this direction. I assume that it would support a form of nominalism, but this should be more fully explained.
Another project of a philosophical nature concerns the foundationalism-coherence debate. If we employ relevant similarity instead of truth-presenvation as a means o evaluating inference then the traditional concept of justification, if it must not be abandoned altogether, must be radically altered. This sheds a completely new light on the traditional problem and is worth investigating.
TNP: 20 Years On
[83] Rumelhart and MacClelland, Explorations.
[84] For example, we may switch back and forth between views of a Necker Cube.
[85] J.A. feldman, "A Connectionist Model of Visual Memory", in Hinton and Anderson (eds.), Parallel Models of Associative Memory, p. 51.
[86] See William James, The Principles of Psychology, p. 279.
[87] Like Nelson Goodman.
The Network Phenomenon: Empiricism and the New Connectionism
Stephen Downes, 1990
(The whole document in MS-Word)
TNP Part X Previous Post
XI Projects and Investigations
A. Computational Difficulties
Let me now conclude this paper by outlining a number of areas of further investigation which ought to be pursued in order to accomplish the fullest and most useful presentation. These areas divide into two distinct categories: conceptual difficulties and computational difficulties. Let me outline some computational difficulties first.
By computational difficulties I mean aspects of the implementation of connectionist theories on computers. A number of concerns can be raised by viewing connectionism within a philosophical framework and some additional features are required. What I would like to do here is actually build a connectionist system using the C programming language and intended for application on an IBM XT compatible or clone. The large number of options, for example, different learning rules, will be incorporated as options on my own system. This system will fill a void on the market: an easy to use connectionist system which costs less than $1,000.
Having developed a connectionist system (which I'll call SDPDP) I want first to look at network variability in aconnectionist system. First of all, I want to construct nets in which different options may be employed in different parts of the same net at the same time. For example, in a PDP net [83] either every unit employs a stochastic on-off activation or every unit is activated in degrees. But in some systems, we want to be able to have units of both varities. In addition to variable structure, I want to incorporate some mechanisms of network plasticity. For in human systems, not only the connections, but the units themselves grow in response to input, especially in early life. Finally, I want to consider what I call "dimensions". For we want it to be the case that such things as religious conversions and scientific revolutions are possible. This requires that a network be able to construct [alternative] pairs of stable representations at the same time, which may alternate in priority. [84] Each of these alternative representations I cann a "dimension".
Another computational problem which I wish to consider concerns learning schedules and annealing. Currently, PDP systems employ a system which is very similar to that employed in physics. But, first, it is not clear that an annealing equation which is suitable for thermodynamics is suitable for human brains. I would like to investigate grounds for choosing one, rather than another, annealing equation. Second, it is clear to me that the annealing schedule employed is inadequate. In my view, temperature increases and decreases ought to be cyclic, as for example paterns of increased brain activity when we sleep. In addition, temperature ought to be sensitive to input, so that we can rapidly process conflicting input.
Finally, there is the hardware itself to think about. Human hardware is much smaller and more complex [than] contemporary computer technology. Perhaps we will not be able to build actual neurons, however, it seems reasonable that, now that we know exactly what we are looking for, we can make some plausible suggestions regarding how to build a computer neuron. I think that it would be best if many of the features currently represented by parameters, for example, threshold or rest values, can be implemented physically.
B. Conceptual Questions
By conceptual questions, I mean investigations into some of the things which connectionism can tell us about epistemology and the philosophy of mind. For, if the arguments concerning rules and categories are sufficiently strong, then we will want to reevaluate such concepts as knowledge and belief. For example, I would like to say that an item of knowledge is a stable pattern of activation, a pattern which tends not to change given varying input. If this is the case, then I may want to say with Feldman that "you do not have a store of knowledge, you are your knowledge." [85] In such a case, then, it becomes necessary to explore what we are and what part of us it is which is our knowledge.
In addition, I want to consider questions concerning theoretical and physical parallelism which arise. For example, through this paper I have used the terms "neuron" and "unit" roughly equivalently. I have also talked of the advisability of using this or that learning equation according to whether or not humans actually employ (or instantiate) the equation. We need to ask, first, whether or not we should design systems in parallel with human neural structure, and if so, what they would look like, and even further, how we would determine what they would look like.
As another investigation, I want to make some remarks about the nature of knowledge (as opposed to the definition of knowledge). For, if knowledge consists of stable patterns of activation then we cannot think in terms of knowledge as being sentences which have a given propositional content. It is unclear whether we can assign propositional content to patterns of activation. If that difficulty does arise, then we may want to consider some other relation between that which would serve as content (for example, representations of events in the real world) and patterns of activation. Here, perhaps, one could follow Armstrong and oldman and assert that there is a causal connection (and distinguish between appropriate and inappropriate causes). In order to successfuly defend this approach, it is necessary to give a fulla ccount of how we learn about causes.
Yet another investigation concerns consciousness. I have suggested above that there are conscious and unconscious regions of the brain. My belief is that those regions which are conscious are those which correspond to the activation of sensory input areas. In other words, my hearing someone speak a sentence and my thinking in a sentence is an activation of the same set of neurons (or an overlapping set). This solves the problem of how we can have a "stream of consciousness [86] in a non-linear network. But a much more detiled story is required here.
Finally, it is worth posing the question of whether connectionism is a type of scientific revolution, in the Kuhnian sense, or whether it is not. Some philosophers, for example Stich and Johnson-Laird, have expressed the opinion that it is not. In my own view, since so many traditional concepts must be overturned, then it is a scientific revolution. Haveing said that, however, I must ask whether or not we are working within an eliminativist paradign, as suggested by, say, Churcland, or not. In my view, there is still a role for words such as "knowledge" and "belief". If I believe this, then first I must explain this role, and then show how this role makes sense within the new paradigm.
C. Other Projects
When I began by asserting that connectionism vindicates empiricism, I embarked on a philosophical enterprise. What followed has been primarily technical and non-philosophical. I would like to return to a connectionist treatment of some philosophical issues.
For example, some contemporary [87] adocate a form of nominalism. While the philosophical debates concerning realism and nominalism are periphrial to this project, it is still the case that connectionism, if successful, should shed some light in this direction. I assume that it would support a form of nominalism, but this should be more fully explained.
Another project of a philosophical nature concerns the foundationalism-coherence debate. If we employ relevant similarity instead of truth-presenvation as a means o evaluating inference then the traditional concept of justification, if it must not be abandoned altogether, must be radically altered. This sheds a completely new light on the traditional problem and is worth investigating.
TNP: 20 Years On
[83] Rumelhart and MacClelland, Explorations.
[84] For example, we may switch back and forth between views of a Necker Cube.
[85] J.A. feldman, "A Connectionist Model of Visual Memory", in Hinton and Anderson (eds.), Parallel Models of Associative Memory, p. 51.
[86] See William James, The Principles of Psychology, p. 279.
[87] Like Nelson Goodman.
