Jul 11, 2014
For those who are on audio, I want to acknowledge Samuel Noekowsky, who just spoke. This talk is basically what he said, except he said it in the French way with complex theories, while I'm going to give it in the prickly, analytical, English kind of Canadian, dry presentation. But I do want to acknowledge it.
For those on audio, and also for this in the room, this is third in three talks. It forms a third part of a set. I did a talk just downstairs on open content. I did a talk at the London School of Economics on Wednesday about the role of the institution and the changing nature of online learning. This follows from that. This addresses the specific question of assessment in open learning and in informal learning.
Faking Our Way to Success
What we have here is a woman who was a dean at MIT for decades, receiving the institute's highest honor, the MIT Excellence Award for Leading Change, and it turns up she didn't have any degrees, not even a bachelor's degree.
Obviously, she's disgraced, et cetera, and she should not have done that, but the interesting thing here is that she can do an excellent job and be recognized for this, by a major institute of higher learning, and yet not have a degree. It raises the question of the value of the degree itself.
On the other hand, we have the possibility that many of us are just faking our way through. It's not just faking our way through doing jobs without degrees -- although I do wonder how many other people are out there doing that. It's faking our cultural knowledge generally. People acting like they've read "War and Peace" when they really haven't read War and Peace, or people referring to plot elements in "Moby-Dick," like I do, without having read Moby-Dick, which I haven't.
Like Learning a Language
When we get into this sort of discussion, we wonder, is there a core, a common heritage, which we each share, or rather is each individual, as we just heard, a strand running through the fabric of culture?
It's interesting to think of culture in this perspective, as a type of language -- culture as something that helps us communicate with each other. Samuel (Noekowsky ) tried to use Greg Egan (as an example), an attempt that failed miserably. I nodded at this because I'm familiar with Greg Egan, but unfortunately I hadn't read that particular book so I felt badly having nodded. These little cultural misrepresentations.
It's funny, because when he talks about knowledge and language in this way, we have this sense that we're talking about knowing a language, knowing a culture, knowing a whole set of facts, and if we could just get all these facts, then we'd know what people know.
But, in fact -- Wittgenstein here is the expert on this -- a language isn't just knowing a set of facts or knowing a set of rules. A language is something much more dynamic, much more behavioral and complex -- but much more like playing a game.
Looked at from this perspective, what constitutes success? Playing the game, being a good MIT Dean of Admissions constitutes success, rather than a demonstration of a set of facts. What is it for one of us to be a success? Do we count our citations? Do we count the impact factor of the journals that we're writing? Probably we shouldn't, right?
It raises a lot of serious questions about assessment, understanding what constitutes success, what constitutes learning. Here's one question: We have many more ways of finding work today, but it seems harder and harder to find a job, harder and harder to match ourselves to a position. Why is that the case?
Here's another question: today's students leave a lot of data traces, as we've just heard, from demographic information to how they read and highlight eBooks, et cetera. What are the ethics of using this for the purposes of assessment?
Here's another one: everybody's learning experiences are customized. Everybody is, as my wife likes to say, a special snowflake, where their experience exactly matches who they are. They study exactly what they best comprehend. So does this mean everybody gets an A?
Is open online learning or the MOOC, the massive open online course, or informal learning in general doing the sort of thing that we need? According to the traditional metrics, it's not. People aren't completing their courses. They're not amassing these sets of facts. They're not proving their knowledge by taking tests and getting degrees.
Nonetheless, MOOCs and open online learning generally are shifting the definition of education away from its historical roots to a skills based, instrumentally defined enterprise -- in other words, very much like speaking a language.
What We Need
What is it that we need? What is it that we're after? What are we trying to get at when we're doing assessment?
Figure 1. Skills Gap
One way of looking at it is the skills gap. I don't know whether they have that here in Europe. It's a big thing in Canada. The point of the skills gap is you have a bunch of unemployed people. You have a bunch of open positions. They don't match. The skills the positions need aren't the skills that the unemployed people have.
Trying to bring people to the point where they can qualify to fill the jobs. That's one way of looking at it.
Look at it this way. It would make sense for employers to just go out and say "we'll educate people for the skills that we need, and then we can hire them," but this isn't happening. One of the reasons, I think, is employers don't actually know what these skills are that they need.
They know that they don't have the people. They know that the unemployed people don't qualify for the job, but they don't know exactly what it is that would qualify for the job. It's just when they do the interview, when they have a conversation with the person, it becomes obvious that they're not qualified. An interesting phenomenon.
What's being recommended? What has our government and business community come up with? Well, in Canada we have a thing called the Canada 2020. It's a business oriented group. They're recommending a learning outcomes assessment program, a council on skills and higher education. We actually used to have one, called the Canadian Council on Learning. Now they want it again.
They want education and skills for aboriginal peoples, whatever that means. They want to "narrow the skills gap between men and women," again, whatever that means. They're looking for credential recognition and skills training for immigrants.
I don't think any of these address the point. I don't think any of these address the problem. The problem is how do you recognize, first of all, what you need in society? And secondly, what people actually have in society.
Doing It The Hard Way
This sort of approach is trying to paper it over. They paper over the cracks with committees and lists of skills, et cetera, but it doesn't get to the core of the problem.
It's doing it the hard way. Compiling learning task inventories to define sets of learning activities related to skills.
All kinds of work has been done in that area, although we find that when people actually use learning task inventories, what they're useful at is telling people what they don't know. It doesn't help them get to where they do know.
It's the basis for instructional design. You start with what you want them to learn, design an experience, cause them to learn it, build in some checks, and see that this has all happened in the end. A very traditional approach.
That's basically what these councils and these skills programs are going to do. That's what PISA did. That's what PISA did for 15-year-olds. The PISA task, for those of you who aren't familiar with them...You take 15-year-olds around the world. You give them a standard set of test questions (interestingly, test questions not based on the curriculum of what they learn but rather test questions based on some definition of what the organizers of PISA think they should have learned by the age of 15. That's true.)
Apply these tasks, and then you get a league table. You have Finland at the top, Shanghai -- which for some reason qualifies as a country -- at the top, all the way down to "not us" at the bottom half of it.
Interestingly, the countries that are doing well in PISA are beginning to doubt the whole program. Countries like Finland probably always doubted the whole program. We have Shanghai, and China generally, looking at whether they really want to focus their efforts on this anyway. Their skills and qualities, they say, should also be acquired from a variety of activities, not just studying and testing, such as play, online activities, and games because they understand that knowing isn't just about getting a set of facts that you can measure on a test.
Why are we emphasizing the test? Here's one theory. This is the slide for any who are skeptics.
The per-student cost for testing is currently about $31 a student. Multiply that by 50 million students in the United States, and you get lots of money. Interestingly, organizations like "The Washington Post," who promote testing, also run testing agencies like Kaplan.
Yet, education is still crucial for economic development, for personal development, and there is this sense in which skills build on skills. There is this sense in which education is kind of cumulative. It almost feels like we're piling facts on each other, even though we know it's not.
We have reports saying the time in school spent by a country's children is directly related to the productivity of its workers. That appears in the publisher's Pearson's White Paper. I'd probably say it differently because I'm not so concerned about the productivity of its workers. The point nonetheless remains the same -- more education means happier people and a more developed society.
Outcomes and Process
The problem is we're confusing the outcomes of education, the test results, et cetera, with the process of education. That's why it's significant -- that it talked about -- the time in school as opposed to, say, the test grade. If they said the test grades are proportionally related to economic development or productivity, I think they'd have a much harder time making their case.
Outcomes are hard. I read this thing from Pearson this morning, "You can be certain and vague, or you can be precise but not certain. You can't be both." You can be both, but you can only be...Let me refine that. No, never mind.
The really interesting, useful outcomes like "understand" and "appreciate" -- so much is built into the word appreciate -- they're pretty much undefinable. They are, in a certain sense, ineffable. We could not really express what we mean.
This is what Samuel (Noekowsky) appeals to, the ineffability, and things like that. The language actually fails us because we think of the language as a language, as a set of facts, structures, rules, et cetera, and this doesn't work.
There isn't a nice, neat set of concepts and principals linked to what we mean by "understand," so you get behavioral outcomes, "display," "recite," "define," but these are based in rote. That's the problem with these tests -- you can fake them.
"The complexities matter." Gardner Campbell. "When confident, simple, plain orderly advice is given about a subject matter," he says, "I hear the sound of the hatchet replaced by the sound of wood snapping as the branch I'm sitting on gives way."
It's easy to recommend simple outcomes-based ways of developing education, but as soon as we begin to do that, we're undermining the foundations of the educational system. This is the dilemma. One the one hand, all of that is true. On the other hand, our knowledge is in here, and what's in here is incredibly difficult to access.
Black Boxes / Making Stuff Up
Even if you slice open the brain -- and people have. I've heard this -- you still don't see knowledge. You see gray, messy, gooey stuff.
What are we going to do? Do we continue using the black box approach, which is based on "recite," "define," et cetera, or do we start making stuff up?
This is where I was worried about Samuel (Noekowsky) for a bit. I was worried he was going to go into the land of making stuff up. When he started talking about mathematics and dust, I sort of worried about that, but he's way better than most.
Here's an example of what I mean by "making stuff up." This is someone who I won't name for his own protection. "Our brains need some way of deciding what to encode and how to encode it, so as to retrieve it in a way which is useful. Our minds solve this problem by encoding information along the affective context. No doubt you have studied many papers that sound like this.
This is made up stuff. It seriously is made up stuff. What is the affective context? I know, you can give me a definition, but can you give me a definition that is operationalizable in any meaningful way? Can you give me a definition which is observable in any meaningful way?
When you start saying things like "the brain needs some way of deciding," how do you know this? Is the brain sitting there and saying "oh, my, I need to find some way of deciding how I know"? No. It's a hypothesis. It's making up some kind of activity the brain is doing, but the brain is not actually doing it.
Male Audience Member: How do you know that?
Stephen: We know that because there's no way of knowing that.
Here's Stephen Talbott. This is a wonderful paper. If you do get the chance, you should read it. We have, even in biology -- which is what makes it shameful -- this tendency to represent the human body, human systems, including thought, even things like the circulatory system, as though they were machines, when manifestly they're not.
Figure 2: The Heart as something other than a pump. Image: https://emedtravel.wordpress.com/page/16/
He has a wonderful account in this paper about the heart and how we think of the heart as a pump. Very mechanical, right? That just probably is the idea that we have in our minds, right? The heart pumps the blood through the circulatory system -- except that if that's really what happened, the amount of pressure that the heart would have to exert on the blood would explode the tiniest veins and arteries in our bodies.
What really happens, he says -- and I don't have a good ground to disbelieve it, and it is well-referenced -- is that our circulatory system is more like the tides. It involves some liquid in the blood system but other liquid elsewhere in the body. It's like the tides that sway back and forth. The heart is a regulatory mechanism, not the sole thing that makes the whole thing work.
An example of this is that your veins in your legs actually play a role in moving the blood back to the heart. The whole body works in this way. The whole body works together as a mechanism for making the blood move. It's not simply the heart. The heart is just one part of it. It's a complex, messy -- really messy. Open a person and you'll see nothing but messiness -- totally integrated organic system, not a machine.
Same thing for the brain. And that's how I know that.
Competency-based Education
That takes us to the next part of our discussion, competency-based education, which is the big thing these days. Competency-based education, as you know, offers credentials based on demonstrated proficiencies. Critics argue that it seems too much like training and it focuses too much on outcomes.
My response to it is it takes a hard problem (whether somebody is, say, a dentist) and breaks it into smaller problems, but all of which are just as hard as the original problem. We've got one hard problem. We've created 10 hard problems.
I wonder, indeed, whether personal learning even requires competencies. Do we require competencies in order to learn? Is a discussion of our learning at the same time a requirement for a discussion of the competencies that we've acquired? In one sense, yeah. You can't really do personalized learning without common expectations about competencies.
Figure 3: Competency Pyramid (cf. Dale's Cone) Source: op. cit.
But this is only under certain circumstances, where we require common expectations as an outcome of personal learning. What if that wasn't the case? Maybe it isn't the case. According to this study from a competency-based education from the Higher Education Quality Council, there's no systematic comprehensive study indicating that the purported skills from competency-based education translate into performance.
This Charles Ungerleider and his crew...I've had my criticism of Ungerleider in the past. I still do, but that's an aside. He is, nonetheless, correct about it. What are the competencies underlying performance in anything? What are the competencies underlying a satisfactory high school education, even?
We think we've got a good handle on what these basic competencies are, but we don't. We think we do, but it's an illusion. I remember so distinctly sitting in an auditorium in the Southern United States. A man got up and said, "We all know what the basics of a fundamental education are. They are mathematics. They are language, and they are religion."
Alternative Credentials
What's happening now? What are we getting? We are getting alternative credentials. We used to have a system where we had high school diplomas, bachelor's degrees, master's degrees, PhDs, and a few professional certifications. Now, because there's so much information out there and so many ways of interacting with people, we are getting a ton of alternative credentials.
As I say here, a veritable slew -- Qualt, for example, based on courses developed by the Association of Accounting Technicians, et cetera, and brought to us by the good people like Google. Harvard has created the Credential of Readiness, which means that you are ready to take a Harvard education, I guess.
ALT, the Association for Learning Technology, in Britain, is designing and creating badges as part of its ocTEL MOOC. On top of that, the badge issuing system, which can establish whether the badges you are issuing are compliant badges. So we have a credential for a credential.
Udacity, Sebastian Thrun's company, together with AT&T and 1.5 million of their dollars, is launching something called nanodegrees. In Europe, the European Commission has the VM Pass, a validation process based on a combination of peer review and crowd sourcing.
Condé Nast - yes, the magazine publisher - is issuing college credentials. The experts are going to be the writers and editors from various magazines. Of course, the publisher will provide some financial backing. It actually convinced some colleges to go on with the scheme.
The Achievement Standards Network is offering "open access to machine-readable representations of learning objectives published by education agencies and organizations including the Common Core State Standards." Recently acquired by Desire2Learn.
Learning Locker is an open source Learning Record Store for tracking learning data.
What Are We Validating?/ Understanding
The Holy Grail is finding a sustainable and reliable method for the validation of non-traditional learning.
But -- and this brings us back to Samuel (Noekowsky) -- are we validating the learning, or are we validating the person? Who are they hiring when they hire someone, the learning or the person?
The objective was, remember, jobs, skills, gap. They don't hire learnings. That's why they don't just use the CV when they're hiring people. They're hiring the person. You wonder why they would do this. They actually go through a process of selection, where real people sit down and talk to job candidates. Why does that happen?
We need a basic understanding of understanding. This is part of my criticism of portfolio approaches. When you're trying to show that you understand something, the answer to that isn't really "let me demonstrate." The reason why "let me demonstrate" isn't the reason is that it's too easy to fake. If you just show a production of some sort and say "this proves that I know of quantum physics, because look," people aren't going to believe that.
There is a sense, though, in which understanding quantum physics is more about doing stuff, as I mentioned at the top of this presentation, than some mental state, some set of facts in the brain that the brain needs to remember.
To know something -- and I've said this many times -- is to recognize. I mean that in a very precise sense. What is it to know something? How many of you are familiar with the books, "Where's Waldo?" Somebody says it's a different name here. Wally, "Where's Wally?" Thank you.
Open up "Where's Wally?" or "Where's Waldo?" for those listening in North America, Africa, or Asia. You look and you search and you search. Finally, you see him. You recognize him. The next time you do the same thing, it takes you no time at all. There he is. These things only work once.
Knowing is like finding Waldo. It's being able to recognize him in the sense that you can't un-know it once you see it. Knowing is like picking out the face of your spouse or your close friend in a crowd coming over in a train. Easy to do. Incredibly complex process, but easy to do. You can't not do it. That's the key.
It's a physical state. It is quite literally the organization of the connections in our brain. Our brain is a perceptual mechanism. It's a perceptual mechanism such that if we see something that we recognize or we know, we get a cascade of neural activation, and we get this characteristic "Ah, there's Waldo. Ah, there's my wife. Ah, there's a tiger." You can see how it would be useful.
Knowing and Doing
It's "to do," in other words, rather than "to know." This is important. When we're doing these assessments, when we're doing these measurements, we're after facts and principles and things like that. Theories. Even language. Even a definition or description.
This is a representation or a model of what we know, rather than what we know. It's a tool, an aid to learning, an aid to understanding and comprehension, but it is not literally the thing that we know. The thing that we know literally is this recognition process.
How do I know that? Because our brain is a pattern-recognition machine, not a digital computer. It's not even a pattern-recognition machine. It's a pattern-recognition whole bunch of gooey complex stuff, interconnected neurons, not a digital computer. The digital computer analogy is simply wrong.
Male Audience Member: If that [inaudible 31:49] how can you do without [inaudible 31:57] ?
Stephen: That's easy. Did you attempt to demonstrate something by asking that question, or were you just asking the question?
Male Audience Member: Good question.
Stephen: You see the distinction. A demonstration is a requirement intended to satisfy a specific set of criteria. But to do is everything. It's a good question, and it does lead into the next bit here.
Here's the secret formula. It's not really a secret formula, but there's a blog over there called, "The Rapid eLearning Blog," which is all based on "do a few videos, and learning will magically appear." That's not true.
But this bit is true. How do you become, or establish yourself, or show that you are an e-learning professional or anything else? First of all you practice what you do. Secondly, you show examples of what you do. Thirdly, you show what you do and what you learned.
Not just the examples. You show the process. You show the thinking. You show what goes on behind the scenes. That's very much how I've structured my career, and for me it's worked reasonably well.
Figure 4: My LinkedIn Network
Here's me showing what I know. This is my LinkedIn network, and it shows a lot of connections. You can actually analyze what the different colors mean. Most colors were actually put in by the computer. I didn't do it. The computer autodetects clusters.
For example, this green cluster here which sticks down is not India, even though it looks like it, is... They're all connected to me, but they're also all connected to each other. It forms a natural cluster. Since it forms a cluster, the system recognizes "hey, that's a cluster. Let's give it green," so we have green.
Automated Assessment
But me showing what I know is me being on this network doing what I do, whatever I do. All of those people see it, and they talk about it. Maybe they talk about it, or maybe they ignore it. Who knows? They do what they do.
This is the wave of the future. There's a system called SCROLL, a System for Capturing and Reusing of Learning Log, and basically it's capturing your learning as you go along.
Think about what happens when we add learning to the Internet of Everything. The Internet of Everything is the Internet of people, of resources, and of things. We're hearing about the Internet of things, but what's really interesting is the Internet of everything.
I used to talk, many years ago, about this fishing rod that teaches. All you do is you take your fishing rod and go out and fish, and the fishing rod has internal sensors that detects how you cast. Then it remarks to you, "You've never fished before, have you?" or something like that.
The fishing rod gradually teaches you, but what's more interesting is that the fishing rod and you are interacting together. It has sensors. It's detecting what you're doing, and it's using that feedback to help you learn.
It was always theoretical, but a month ago I saw a TV show. They didn't do a fishing rod. They did a tennis racket. People who buy a tennis racket will pay more than people who fish, I suppose. I don't know why they chose a tennis racket. Maybe there's more room in the handle for the electronics.
The idea here is that as time goes by, we and the machines have to get skilled, are getting skilled at lumping data and things together and then filtering and understanding the language. All of this data basically tells, as Samuel said, the story of who we are. The story of what we do.
It's assessment based on public performance. It's like an essay, and it can be assessed in exactly the same way an essay is assessed. The way an essay is assessed now is a person reads it. Sometimes they'll use a rubric but again, that's taking one hard problem and creating 10 hard problems out of it.
Other times they'll look at it, and the mechanism by which they determine whether it's an A or an F -- and they've decided that probably very quickly – maybe even after the first few paragraphs. Certainly when I read papers -- and I read lots -- I force myself to read the rest of the paper, but really I know.
You know because you can tell. You can recognize it. See how the words are used? See how the sentences are constructed? Are they even addressing a real problem? All of these things are in a certain sense ineffable.
Like Dreyfus would say, "The knowledge of an expert is intuitive. It's recognized." The machines are getting smart enough to do this themselves. Neural-network software is getting smart enough to do it itself.
That's how they mark essays. They don't mark essays using key words or grammatical constructions. That's a popular fiction.
If you look at the actual way they do automatic grading of essays, they give a neural net a training set of five, 100, 1,000, whatever essays all previously marked, and then that creates the recognition mechanism in the neural network, specifically the pattern and connection. Then when they put a new essay into the system, it comes out with a grade based on that previous experience.
Combine two things, the mechanism that creates clusters like we saw on LinkedIn and the mechanism that can automatically grade essays, and you have the potential for a system that doesn't actually need to be trained. You just let it out there into the world, let it find expert discourse for itself, and then it can associate new discourse with the expert discourse.
A Recognition Task
I've just waved my hand here at what is in fact a very difficult and challenging problem. We have researchers and engineers working on this problem...Not easy. But we know the machines are getting closer.
We know this because machines are passing the Turing test. The whole thing of chatbotwhich has been going on since the days of MUDs, IRC, and I think Julia was the name of it, where people would have conversations with these bots...That's been continuing since then.
Grading is a recognition task. It's what neural networks do, and it's what we design and build interfaces for ourselves to do. It's how we'll respond to the Internet of things. We're not going to examine all of that data personally. If we could, we'd design systems that present to us maps, graphs, dashboards, things like that. It's how we're beginning to understand the world now.
Sometimes it's talked about as intuitive. Sometimes it's talked about as a language. Sometimes, Don Tapscott, Marc Prensky, and the rest would talk about it as a culture and a generation in that. That's what they really mean -- beginning to understand the world through this process of recognition.
Preskett and Prensky might know that this is what they're saying, but this is what they're saying. It's not a generational thing. It's not a "ooh, we can all multitask." It's that if you look at this wide set of data you can see. You can look at multiple thread streams, multiple Twitter feeds at once, and see the pattern.
How do you see the pattern? Because you train your brain over time. You have been exposed to this data in the past, and it creates a set of connections. Your brain is a pattern-recognition machine. You see Tweets, LOLcats, whatever, and you recognize them like you recognize Waldo, like you recognize a pattern.
It's an instinct of knowledge. It doesn't have a name. It doesn't have a word. It doesn't have a concept. But it's a thing. People know what it is.
We're beginning to become sensitive to these tunes, to these signals, sometimes even overly sensitive, sometimes even hypersensitive. Our reaction to some of the stuff that goes on online is proof of our recognition of some of the stuff that goes on online.
What We Reveal
On the one hand, we reveal ourselves in our messages. We reveal our thoughts, maybe times a lot more than we intended. Sometimes these assessments, what we reveal, what other people look at, can be very personal, can be very uninhibited. I think we know this.
We're getting sometimes some very brutal -- not necessarily honest but nasty -- assessments online. There's the whole range of reactions. The range of reactions is from the very positive to the oh-so-very negative. I've had both, and I think we all have had both.
We're seeing all of this in other areas first. We're seeing big data used to analyze all of these Internet reactions. We have a project at the National Research Council where they looked at the emotional significance of Tweets. You have an event. You look at all the Tweets. Are people generally happy with it? Are they upset with it? et cetera. It's a project. It's done all by neural network analysis of this published data.
Figure 5. Cat with a cone on its head. Source: http://www.sharenator.com/image/10288/
Sometimes what we reveal is very involuntary. You can sort of get this reaction. You all saw the poor cat there with the cone on its head, and you probably all said "aw." This "aw" sensation is revealed in your later communication. Sometimes you actually type it out, "Aw."
It's interesting because it's not just a one-way thing. There's a dynamic interplay. I was thinking about this. There's of course this whole literature about violent video games and that. I've played violent video games. I don't think that they desensitize you.
The reason that I don't think they desensitize you is that there's no real emotional attachment to the figures in the game in the first place. It doesn't trigger the apocryphal mirror neurons. They're maybe not that apocryphal, but you know what I mean.
But I'm also a devotee of fail videos. That's a little secret. But I love them because I see myself in those. You see somebody on the bicycle going over the handlebars, and you go "ah." You do have that feeling. That feeling is what comes out, and that feeling does get reflected. Then Facebook analyzes it and monetizes it.
These assessment mechanisms are being built into the LMS. You require your students to use an LMS. They use an LMS. Your system now begins to analyze them in more and more and more detail over time. They probably won't stop at the LMS, will they?
Raising Questions
People are beginning to raise questions about this, and I think these are important and good questions. They're asking about the methods of exploring the types of data. What kinds of data is appropriate for an institution to collect? Should an LMS be collecting information that reveals your emotional state? They're asking about data fishing. "Let's see what's there."
The worlds of privacy and the worlds of analytics interact. This is the problem with traces. You leave these traces, and they're analyzed. People see into your soul -- or other non-made-up thing -- do we have the right to do this, or do we have the right to, as Samuel said, take back our traces, take back our digital presence?
In a certain sense, that's impossible. It's like me trying to take back the fact that you all saw me 10 minutes ago -- "I'd love to take that back. It wasn't one of my better moments" -- but it's not possible.
You have heard about this one. I just referred to it. Facebook is doing an unannounced experiment on the emotional reactions of 689,003 of its users to show that emotional states can be transferred to other people. They messed around with the lists of stories displayed in their news-stream or their feed and then measured the responses to detect emotional responses to that. Intuitively it's measuring something that we understand does exist.
You show people nothing but stories of crime -- and there have been studies on this -- and they think there's a high-crime rate. Even though the crime rate is going down, because all they see is crime, they think it's going up. Facebook was doing the same sort of thing, except they didn't tell anybody they were measuring this.
You've probably heard about the reaction. People want to get out of Facebook, and they'll jump to Twitter. But it's not just Facebook. Yahoo -- among others, I think Google is the other one -- is dropping the do-not-track mechanism. That's a signal you can put in your browser so that people don't track you.
They're saying that it's too confusing, but they're the ones who made it confusing. They're the ones who established the signal in the first place.
Google announced last April that they had halted the practice of scanning student Gmail accounts for potential advertising purposes. Wait a sec, that means they were scanning student accounts for advertising purposes. They've decided to stop acting like your creepy uncle. I feel much better now.
Sometimes it's accidental. Sometimes the data just gets out, but when it gets out it can be a bombshell. Like this one, "University of Virginia Law School collecting and distributing to potential employers information about grades," OK, "class ranking," I don't know, "political affiliation, work experience, recommenders, information about where their girlfriend lives." This is secret data. Students don't know about this, the employers do.
Options
What do we do? One option is to delete all of our social media accounts, but we're not going to do that, are we? People are not leaving Facebook. They're not leaving Google. They're not leaving LinkedIn. It's not going to happen. These services are too useful.
We learn about our friends. We learn about ourselves. It gives us this mechanism by which we can recognize what the state of affairs of the world is. So we're not going to leave the system. At the same time, companies are beginning to feel the heat.
Figure 6. InBloom. Source: op.cit.
Gates funded a thing called inBloom. It was based in Atlanta, and basically it was "we will centralize and store all of your student data for you." Well, we know about that sort of practice, and we have Rupert Murdoch on the one hand, Gates and Carnegie on the other hand, and Joe Hacker on the other hand all accessing this data.
Of course it raised a substantial response, a substantial reaction, and inBloom was eventually shut down, as it should be. Ironically the new concern about data is called the Snowden Effect. That's what you classically call shooting the messenger.
But it's true. Canada just passed an anti-spam law, which I can report in my case did not change the flow of email into my inbox. There was a surge before the law was passed of people asking whether I gave them permission to continue sending messages to me. I took great delight in not giving them permission.
As of July 1st, the same messages, the same companies...The flow continued uninterrupted. But there are laws in place now.
This whole concern about privacy is spreading -- and quite rightfully so in some cases, wrongfully so in other cases. Lecture capture...You'll notice I did advise everyone that I was recording this ahead of time. I hope it's still recording. Oh yes, it's still chugging away there.
People are demanding that the classes they attend not be recorded because they think that the classroom is a space where you expect privacy. It's an interesting question.
But it's also becoming more and more the case that when you attend a class, cameras will be on, if not officially at the front of the room, unofficially in the back of the room. I could give you a whole bunch of neat little examples of teachers rampaging on, but I won't.
One proposal -- and this comes from Doug Belshaw with Mozilla -- is to ensure that common spaces are public spaces and not privately owned. There's one weakness in that proposal -- our public spaces aren't very private either.
Personal Privacy
It's not simply the case that we can take these services out of the hands of the billionaires because if we put it in the hands of the government, we get, again, the Snowden Effect. The real answer here, I think, is personal privacy. Personal privacy and informal assessment are going to go hand in hand. They will necessarily go hand in hand.
The schemes -- and you will hear a lot more about these -- big data and learning analytics are going to flounder on the rock of personal privacy. If you haven't heard it before, you've heard it here first. I specialize in topologies.
There's clear indication that people want this. One example is on a crowdfunding site called Seedmatch raising money for an NSA-proof personal server. They asked for...I forget the exact number, but it was something like €75,000. In 89 minutes, they raised €750,000.
People are moving to privacy securing personal networks. There are a whole bunch of them. Privatext, TigerText, Whisper, Cyber Dust, Ansa, Omelette, and Diaspora. Personal disclosure, I invested in a whole 100 dollars in Diaspora.
A New System of Assessment
Let's put all of this together, wrap it up, and tie it up in a neat bow. The elements within your system of assessment are going to involve personal servers. Not stuff you store on the learning management system, not stuff you store on Facebook or LinkedIn, but stuff you store on a network you control and own.
There's going to be a public space, the information you are willing to share with other people, your friends, the assessors that are out there in the world. That'll be your public face. It'll be the clothes that you put on.
There will be identity management. Us taking control of our own identity.
One of the things about Facebook that's depressing is that -- and this is Mark Zuckerberg's big objective -- everybody will have one and only one identity. It's Facebook taking control of identity over your heads.
One of the things we will see is people can have multiple identities. They can and they will have multiple identities. Why? Because they always have. All of these things are being developed today. Many of them are being developed in our own projects in our own labs at NRC.
Figure 7. Personal Production
Personal production. All the stuff we do online publicly feeds into these content networks. These content networks act, in the first instance, as a global content filtering system, a global perceptual network. You think of the social network as an interconnected network of people. It functions in the same way as a neural network, in the sense that it is a pattern recognizer.
Male Audience Member: Who are the curators?
Stephen: People. I don't want to call them curators, necessarily, because I think people do more than just filter. They interpret, revise, remix, re-purpose, spin-roll, fold, et cetera. Right? More than just curation, although curation is a word a lot of people like to use. It's people, right? It's one person doing things, whatever they do, and these things are seen, recognized, passed on, commented on, et cetera, by other people.
The first layer here is the social network. That creates the mechanism for assessing the qualifications of the individual. Picture it...You can think of it...I'm trying to draw a picture now, and I don't have a slide for it, which makes it hard to do. Imagine a social network of quantum physicists. We know all the major quantum physicists know each other. They communicate with each other, respond to each other's papers, go to the conferences, call each other names, et cetera. There's this cluster, just like the cluster of green people in my LinkedIn network, except they're quantum physicists and therefore have no color. They're linked to each other.
My qualifications, as a quantum physicist, can be mapped and understood as my positioning in this cluster. If the other quantum physicists talk to me -- this is putting it very crudely, of course -- I rate as a quantum physicist, but I can't fool the other quantum physicists because the one person who recognizes a fake quantum physicist is a quantum physicist.
This human network acts as a perceptual mechanism for filtering the qualifications of people. That's why the person could fool MIT staff. The degrees actually don't matter. The way you relate and interact with other people...That does matter. That's what's happening online today.
As an aside, it has always happened. That's the way it's always worked except we didn't have a global communications network to pull it off before, so it was always very local, very individual, very personal.
Professions will coalesce around this. They're online communities. They're open ended networks. They're similar to vendors' communities in practice, but they're not just that. They might be gaming communities.
I got my programming chops in a gaming community known as a MUD, Multi User Dungeon. This is a game, "Untrusted". Same sort of thing. It's really neat. It's a maze where you solve each step of the maze by editing the JavaScript in the window. That lets you through the maze.
All of the artifacts of our interactions with each other can be measured and understood in terms of their impact on our qualifications and credentials. Conversations can be analyzed for meaning and word use.
Interviews. Instead of a person interviewing with 10 different companies, a person does an interview with a professional interviewer. The video and transcript are made available online, and then these 10 companies can look at that interview, use their system to assess that interview according to their internal criteria, and determine whether or not you get the job. It's a first-order screening , but it's a very effective one.
Would you hire him? Could a machine detect whether you would hire him?
Assessment of the future will basically redefine body of work. It used to be formal publication, books, paintings, films, whatever. Now it will be the mass of communications, my mass of activities, whatever we do on the public Internet.
That will be a good thing because it will mean that we're able to obtain a much more accurate, much more precise assessment of people and it will help to map, first of all, the gaps in knowledge that need to be filled by learning, which is one of the purposes of assessment.
Secondly, it will help companies, employers, contractors find and identify the people who most precisely fit what they're looking for.
That's my talk. I thank you very much for your time and attention and even occasional scepticism.
[applause]
Male Audience Member: Stephen, I have to believe there's something wrong [inaudible 60:13] talking about [inaudible 60:17] . Do you get that the biological understanding of the inputs is the right [inaudible 60:41]?
Stephen: The short answer is yes, but we need to be careful about how that cashes out because, as I mentioned with the reference to the once slide with Steve Talbot, what our understanding of biological systems is needs to be clear, as well. It needs to be, if you will, true to that subject.
A lot of people talk about knowledge, the mind, et cetera, as an ecosystem but when you push them for what they think of as an ecosystem, they say "well, it's a system, just like a really complex machine."
That's not what I mean. A biology, an ecosystem isn't a mechanical thing. It's an organic thing. It's the difference between -- to use perhaps the often overused distinction -- complicated, which has many parts, and complex, which has multiple simultaneous interactive variables and where the outcomes are not predictable, except when they're probabilistic or stochastic mechanisms.
Male Audience Member: I really like the vision [inaudible 61:54] orderly control over the outputs [inaudible 62:16] while they're in their [inaudible 62:19] .
Stephen: Yeah, two very separate issues, both of which are important. First of all, as a management, you want personal identity. One of the reasons why Facebook succeeded -- let's face it, they succeeded -- is because they made something that was originally pretty complex, interacting with thousands of people online and keeping track of that more or less, and made it pretty simple.
It replaced a previous very useful system, called thefacebook, which literally was a book of faces, only now it lets you talk to them as well. This kind of system where we manage our own personal data will become easier over time. Some of the things that I referred to -- the personal web server and things like that, for example -- are things that you will be able to just buy as a box from Future Shop or Media Markt. It will just be a box. It will cost about $300.
You put it in your living room. It'll connect your home network, which you already have because you have Internet access probably. Then you access it the way you would access Facebook. It'll be that easy.
There's open source software like ownCloud, for example, that already allows you to do this. Set up a cloud server, just like Amazon Web Services or Google Drive, using your own cloud and set one of those up on a personal server in your living room. It's as easy as using Dropbox or any of the others.
That's that part. The technology will make it easier. As it becomes easier, the take-up will increase. As the take-up increases, we get these personally managed social networks as opposed to these centrally managed social networks.
It's like the browser. It used to be that the only way to access the Internet was to log into a central server. It really was. Lynx was the first web browser. Lynx was a server application. It would run on your central computer hosted by an institution. You would log on to Lynx through Telnet, sign on through the central server, and then you'd access all the other websites.
The personal web browser goes on your computer. It's yours. You can configure it yourself. It's easy to use. It's a lot easier than Lynx actually. Nobody uses Lynx anymore. Nobody would use Lynx even if it had pictures, which it didn't. You can't really justify that for a central server environment.
The second question about security and player data is a good question. I've done work on DRM, and there are two approaches...Roughly, widely, there are two ways of looking at DRM, which is a dramatic overstatement of the distinction. You can either manage the location where the resource is located or you can manage the resource.
If you manage the resource, then you don't have these kinds of problems. If you share a resource with an individual where the resource doesn't go beyond the domain of the individual. [inaudible 65:56] .
[laughter]
Stephen: As an aside, the short answer is there are technical solutions to this. The longer answer is none of them will work. It's the Microsoft Darknet paper all over again. Ultimately, you can't establish security through technical measures. It depends on the interest of the people involved being better served by preserving the security than breaking the security.
Edward Snowden had a choice, and the choice that he made wasn't "oh, I can break this technical measure." He just walked right through that. It was easy. He was inside of the system. It was "it's better socially, morally, politically, to release this data [inaudible 67:01] ."
People are always going to face that at some point. You have a password for your corporate account. You have a VPN. Why don't you share that with everyone else? You don't. You don't because it's better for you that you don't. You have a bank card. What's your PIN? You're not telling me. Why not?
It's not that the technology makes it impossible for you to tell me your pin. It's that you won't. That's where security comes from in the end. Companies need to learn this. It's easy to think "oh, we'll just set up a wall."
People talk about military grade security. Have you been to Gagetown? Gagetown is a military base in New Brunswick, Canada. It's about an hour away from where I live. Anybody can walk into this military base. Seriously. It's located right beside the town of Oromocto, by a major highway.
There are signs on the highway saying "you are in a military base. Please do not stop." The implication being "because we might shell you." [laughs] You can actually...And I've done this. You drive in, park in their parking lot, get out of your car, walk into the building. You can just walk. Nobody challenges you. There's no lighting, no nothing.
The security isn't in creating a fence. The security is all the bits and pieces. The people inside that base recognize whether or not you belong there. The people inside that base know that security [inaudible 68:46] , and they don't just leave classified documents laying around.
Male Audience Member: [inaudible 68:57] because I think they [inaudible 69:02] and the individual [inaudible 69:07] . They're based in [inaudible 69:13]. They have a vibrant community over there, but I don't know how farther you're going to go add and what you're going to [inaudible 69:20].
Stephen: Yeah, I'm not familiar with them, but Personal Data Ecosystem Consortium.
Oh, I'll keep that in mind. I'll look for them.
Male Audience Member: [inaudible 69:32] because technically [inaudible 69:35] . Part of the [inaudible 69:44] and how can you help to have the resources [inaudible 69:51]. The second one, the employer is the one who has power. How do you...And get the workers creating value, like [inaudible 70:03] employer? Why is it better for the employer to let work through, keep their information [inaudible 70:09]?
Stephen: It's a pretty complex question, yet again. The "who pays for it" is us because we've always paid for it. Facebook offers a free service, but as everyone knows you are not Facebook's client. You are a product. You are what they sell to their real customers, who are the advertisers, marketers, and corporations. That's who pays for it ultimately.
Those of us who are good at negotiating deals will get our employers pay for some of it. You might get a computer from that. You might get a service from them. Just like my employer pays for my computer in my office, my employer also pays for laptops, but not this one.
This is for a couple of reasons, one of which is that my employer wants to give me a Dell with a power supply that is literally this big -- for those of you on audio, I am indicating the size and shape of a brick. I want to carry that nice light computer, but also because this is mine. I configured this the way I want it.
We've got to come up with some sort of accommodation. There are voices even in my own company, the National Research Council Canada, which is a government agency, saying "there should be no data whatsoever that goes outside the corporate infrastructure," but from a practical sense that's impossible.
The world is more permeable. The world must be more permeable than that because you can't run in a completely isolated network and expect to communicate with the rest of the world. It can't happen.
There isn't going to be an easy answer to your question. Ultimately it comes out to power. The power relations are shifting. It's outside the scope of this talk, but the power relations between individuals and employers are shifting.
Right now it's really hard to imagine a case where the employee has any sort of equitable power with the employer. We used to balance that by unionization, but most of the actions are illegal these days because of a shift in the balance of power.
What's going to change things is the capacity for people to organize themselves, to quickly and efficiently find people, individuals, and contractors to fill individual positions. People will be, because of something like this, more mobile, more able to market their works to a wide variety of employers worldwide as opposed to just in your own city or your own plot. That changed the dynamic.
The relationship between employer and employee becomes one more of a negotiation between equals rather than the exertion of a corporate dominance by one over the other. That's what allows for the negotiations to take place regarding the ownership and the management of the data.
I know employers don't like this. People in power never do, and they will resist this -- count on it. But the technological answer as opposed to the social answer will be to try to find [inaudible 73:55] revenue. The social answer could go any direction.
Male Audience Member: Because of the transition [inaudible 74:03] competency based employer [inaudible 74:06] . You should look at [inaudible 74:12] competency standards as of now. Then you could say a map is very useful to travel, but you don't travel them back. You travel because you're going to go somewhere, [inaudible 74:24] the richness of the direction.
Maybe one solution would be to...We need maps. There is nothing wrong with the map. The question is the place of the map within the educational system. It is usually told to travel -- maybe that's OK -- but you should just admit the use of the map just to travel the map [inaudible 74:47] .
Stephen: Yeah. I love maps, so I like what you're saying. But there are maps, and there are maps.
One way of doing a map is to send a guy on a ship out with an astrolabe and a compass, and to draw pictures of the shorelines. That's what people used to do. Very bad idea because first of all, really expensive. Secondly, the maps aren't that accurate.
A better plan is to launch a satellite and take pictures. That is still not going to be the best map in the world. We could probably still do better, and it's still only going to be a map. But it's going to be a lot more accurate and a lot more useful.
When we use a GPS, we use one of the picture kind of maps, rather than one of the hands-on kind of maps because it's more accurate. It's not perfect -- people have driven into rivers following on GPS. They really shouldn't depend on them and believe that they are reality, but they're better.
Same story here with competences. The whole enterprise, sending people out and drafting lists of competencies by hand, is way too expensive and almost certainly inaccurate. We want the alternate mechanism of automatically recognizing competencies.
We actually have a project in our program called Automated Competency Detection and Recognition. The idea here is it's like a satellite photo of an environment -- in this case a social infrastructure -- taking pictures, understanding the connection, et cetera, and that gives you your map. But it's just a map. Once we understand that it's just a map and that it's based on what are actually real human performances...And it's the real human performances -- and responses to them which are equally important.
Male Audience Member: Yes, but there is a list of analyses. You could use them [inaudible 77:04] map. Then that is a social construction, meaning there is some algorithm designed by some engineer, who will be very smart, [inaudible 77:16] . It is not a map with us. It is a map for us. It is [inaudible 77:24].
Stephen: With great tools people will be able to do this. I don't have any particular opposition to this, but look at OpenStreetMap versus Google Maps. Right now I use Google Maps. The reason for that is OpenStreetMap is really detailed in some areas but not too detailed in other areas, because there are no people there, like places where I live.
I live in a province the size of Belgium with a population of one of its smaller cities. There are lots of rooms, and there are very few people living on the roads. None of them are mapping them. That's the problem. Meanwhile, the Google satellite, or whatever it is, passes over once, takes a picture of the province, and we've got our map.
I like OpenStreetMap. I like the idea of a map that's created [inaudible 78:24] by the people, but there are right now certain technological and structural difficulties not [inaudible 78:30] it work. What would I be working on? A people satellite system. That would be cool but hard.
Male Audience Member: ...But when he talked about the human network [inaudible 79:04] , what happens if that somehow [inaudible 79:14] maybe the entire [inaudible 79:18] has some section that's actually [inaudible 79:22]. How do we then [inaudible 79:26]?
Stephen: That's a good musing, and it's a relevant musing. I'm not sure. The human perception works basically on the same principle, and human perception sometimes goes wrong. Sometimes we're color blind. We only see what we're expecting to see. We don't see the gorilla walking behind the crowd. Perception is not a perfect mechanism.
I think the first answer to that question is, we need as a society to understand we've always had these mechanisms. We're [inaudible 80:16] describing [inaudible 80:17] a giant version of a much more rich and complex network of social perception and to what we've add in the past.
The one we had in the past is reporters going out, watching things, taking pictures, and writing about them, publishing in newspapers, and things like that. It's functioned on the whole very poorly. This is probably better than that, but it's not perfect. One of the neat things about the new system is that it's forcing us to question whether our understanding of the world is correct. That isn't something we've been good at in the past.
In one sense, there isn't a good answer to your question. It's harder to be self-correcting. We need to develop the global mechanism of self-awareness, of reflective reflection -- probably should've used a noun there. Just [inaudible 81:21] the adjective. You know what I mean?
On the other hand, it is a danger, but no matter what we do it's going to be a danger. I don't have a better option. I wish I did. I wish I could assure [inaudible 81:42] is perfect and [inaudible 81:43] is perfect, but no, people will [inaudible 81:47] . People will too often [inaudible 81:53].
Ultimately, with the right mechanisms, I hope we can correct for that. We did ultimately learn [inaudible 82:03] cannot fly.
Male Audience Member: My question partially has to do with [inaudible 82:13] .
Stephen: That's right. I didn't trust [inaudible 83:30] .
[laughter]
Stephen: That's a reaction you're supposed to have. It's interesting the way you described that. What you described with the triangulation is an attempt to offer a mechanical solution, a logical, mathematical solution, framed in the name of [inaudible 83:56] to what is in fact a complex thing. When you depict it as this kind of mechanical, mathematical problem, it becomes intractable and pretty much impossible to solve.
That's one of the reasons why I favor a neural network approach. In the neural network, the quality, if you will, of the inference is described in terms of the structure and the properties of the network. I'm going to [inaudible 84:38] .
For example, one of the properties of the neural networks is to seek the set of connections that expresses the lowest potential letter. It's a methodological [audio cuts] , It might be overall good or might not be overall good, but as a methodological principle, you can apply that [inaudible 85:05] past certainly whether or not it is.
That's a fact. What people [inaudible 85:11] are doing right now. But let's go with that. There's an algorithm, a formula, that describes how a network settles into the state of [inaudible 85:27] . The sort of thing you're describing is, what if they fall into a local minimum? A local minimum is a stable configuration so that it appears through outward appearances like it's the position of lowest potential energy, but it's not the best possible.
To give you an analogy might help. You throw a rock into a pond or bowl or something. The water ripples, splashes, and then the water settles. The methodological principle here is the water settles to the point of lowest potential energy. But sometimes you get puddles in rocks. The water settles, but there's a puddle in the rock. That water isn't at the best it could be. It could be lower, but it isn't because it's in a puddle. It's in a local minimum.
There's a mechanism called a Boltzmann machine where -- imagine this -- you can pick up this pond and shake it. You pick up the pond, shake it -- vigorously at first, but gradually, gradually, gradually less and less vigorously. Mathematically you can show that, in general, not every time, the water will splash out of the local minima, and it will all end up in the bottom of the pond where it belongs.
You can do that same sort of thing with a neural network. You shake up the connections, and then gradually...For example, when one neuron fires and another neuron receives signal, it might fire, it might not. That's a probability function.
You make the probability really high so that it will fire. You have neurons firing all over the place. That's like shaking up the network, and then gradually you lower that probability rule so it's less and less and less likely to fire. The resulting configuration is the most stable possible.
Your question amounts to, how do you know that will work? My answer is, first of all, there is a mechanism, and this describes it. It's based on known principles of thermodynamics. It's something that we can observe with our eyes, working in the world.
It's the same way we make metal harder -- annealing. We can test this empirically by building models of this system comparing them to what actually happens. That's as good as it gets. It doesn't get better. That's my answer.
