Oct 26, 2020
Unedited Transcript
Okay. Terrific well hello everyone, thank you. I'm so glad to be joining all of you and and finally participating on in my first epic so I'm I'm excited about this myself so I'm just gonna go ahead and introduce our first speaker who is we I am delighted that Stephen Downs has agreed join us so Stephen is going to be talking about open recognition networks for those who don't know Stephen, 's work he is the with the digital technologies research center at the National Research Council of Canada specializing in new instructional media and personal learning technology, he is one of the originators of the massively open online course and we are just delighted that he is here to speak with us so welcome Stephen hi thank you.
Quick host keeping note how long do I have because I forgotten I have it written down but you have you scheduled for for 60 minutes so and as much time as you would like to leave for discussion sure that's terrific alright so no discussion all right, no I'm just kidding so here's how I'll do this because I do things a bit differently online than in person we do have a number of people who have joined us.
I'm not sure how many people. It's in this 32 so okay. I need to be a bit careful then but I don't mind questions and comments as I go along so do feel free to interrupt but you know 32 so we'll have to be a bit judicious also this table is a bit rocky so that's not an earthquake.
I've just bumped my table all right, so um, and of course and oh right and chat right and is that, Is this by boat we go over to the US well, um, right now you're in Canada and for those of you who are Europeans the terminal logical note here, if you say America you are referring specifically to the United States if you say the Americas you are referring to all of us so but so if you say America and want to include Canada, you should say America and Canada, all right, that's me being a prickly.
80, and 'cause we have to be It's our job. Alright, so I'm gonna share my screen. Let's see if this works first time out. So you should be seeing the nice big open recognitions network slide. I believe that's true if it's not true somebody please tell me that it's not true.
So, I gotta zip no I'm not gonna zip through this but I got a lot of stuff to cover. I'm talking about open recognition networks some of this will be review for some of you some of it won't be review for some of you and even where it's reviewing.
I'm leading into a general direction, so I hope you'll bear with me. So where we're gonna begin is where a lot of these kind of discussions begin and that's with competencies and stackable credentials. Now competencies or a competency model is a detailed behaviorally specific description of the skills and traits employees need to be effective in a job not a very employment specific perspective and a lot of the discussion in this talk will be employment specific but.
You shouldn't read it that way it's just because so much of the literature around competencies is focused on employment models. And the you know, you can have a competency a single thing which is a competency but more characteristically will have a competency model which will describe a position or a rank within a position or have a number, you know, a couple of dozen maybe competencies with definitions describing.
What counts as effective or superior performance in that particular job knowing the competencies for a job helps people select people for that position helps evaluate people in that position. There's a lot of discussion about you know, what are the precise definitions of competencies skills, etc? I'm not gonna get into that as we'll see through this presentation a more precise definition of a competency probably won't be needed.
So the whole concept of recognition revolves around evaluating competencies. When we're evaluating competencies, basically what we're doing is we're mapping behaviors to the competency definitions and then looking at the outputs and then ideally looking at the organizational results, you know, was there a quote unquote return on investment for the training or for the whole competency program, etc.
There's a lot of discussion here. I've got, A. Link to David gays discussion of the overview of competencies their benefits and uses again, I don't want to linger on that early huge literature on that subject my own work in this area began a few years ago. I began back when I was directing the learning and performance support systems program, but it also evolved with my work in an application called grasshopper, which I used have used over the years to.
Deliver massive open line open online courses and in the 2018 version of the course e-learning 3.0. I set up a whole competency structure and basically I aligned the tasks with the learning objectives in the different modules in the learning resource and you're seeing here a screen from grasshopper with a little overlay in the upper right-hand corner of the the different data.
Involved and the idea here is that people would in each module according to the learning objectives and resources of the module the skill required complete a task of some sort and you're seeing here one task, for example create a model graph and then to the completion of that. I attached a badge and badges were also defined as a separate data element in the open online course.
So that's where competencies intersected with the bad system in my course now in general competencies and credentials such as badges are intended to not just stand on their own but to be as we say stackable so we get this you know, this flow this logic model if you will a bunch of competencies we see them the blue circles on the left hand side and they aggregate to.
Form courses or modules and are recognized using badges and then these badges are joined together or collect together in different groupings to give us certificates and a grouping of certificates may even lead to a degree that's an awful lot of infrastructure. And I think as we'll see infrastructure that we might not need so let's look though a little bit more specifically at badges and blockchain.
In the badge infrastructure that I used for e-learning 3.0. I used Badger as a provider and what I did is in grasshopper, I wrote some scripts that connected to the badger API so I would define all my badges in grasshopper fire off the request to Badger and get the result that you see here so in Badger you have to do a few things you have to first of all create an issuer that in this case was me but, It might.
A university might be university professor you might have issuers that delegate issuing authority to other issuers, then you create what's called a badge class this is typically what we would call a badge, you know, a badge like the say in Boy Scouts the Woodman's ship badge right but that's actually a badge class because the badge instances are the specific individual badges that are awarded to people and, Of course each badge that you award to a person is a difference thing the badge itself is an image and then baked into the image is information about what was done about who the person is what the bad recognizes and the evidence for the completion of that badge there's a bunch of badge platforms out there badgers the most well-known but there's a credible badge cracked badge factor BC diploma.
Bester create my mouse credit. Ly make weaves so on I've got a list of 26 in the notes on my slides my slides, by the way are available and oh, I get to test this if you go back to the beginning, you can see the URL of the slides.
At the bottom of the page there www.downs.ca/presentation /. 532, that's the presentation page. For this particular presentation, you can go to that get the slides after every slide has notes with references etc so let's go back to where we were so that's the badge infrastructure so in grasshopper I set up a badge workflow and so here's here's the overall badge workflow that we used the author that is to say the student the course that is to say the instructor and then badger.
And the workflow is sort of what you would expect it to be, you know, the the person or the instructor creates the badge registers the badge then the author would create a post they didn't put their posts on the website and this is on the eLearn 3.0 course, this was an important component of the way I do MOOCs.
I don't have everybody come into the single centralized platform. I have people working in their own environment and then the platform interacts with that environment so they would publish the post on their own website the post would be whatever the evidence is for their badge then I harvest the information about their post using RSS, that's one way there are other ways of harvesting information from websites the badges awarded the reward is registered with bank.
Your badge or publishes the badge and then I used a technology it's a w3c technology called web mention and that would send off a notification back to the original the original place where the person blogged their their evidence or their task to complete the badge and also badger itself sent an email now that's an awful lot of stuff but my focus really is.
On this little bit right here in the center because this is a really complex thing that you know, all the rest of that infrastructure makes us forget and that is awards the badge to the URL. But before we get to that, I'll just talk about the final step in what we did in e-learning 3.0 we also recorded the badge on a blockchain.
I wrote a little toy blockchain it's a real blockchain it works it works in the same way that say bitcoin or etherium works it's just I made the I made the algorithm a bit simpler the the the hashing mechanism a bit similar the proof of work a bit similar and simpler.
So that it wouldn't take you know, 40 hours to process a badge it actually takes about a tenth of a second much easier for me so what you see on the screen there is the record that would be entered into the blockchain the hash is how we chain the blockchain together and there we go pin wasn't pinning for a bit and that's what establishes the verification for the person having earned that badge in that course.
That time so that's the infrastructure so let's back up a little bit. What we're seeing out there in the world and this is probably not a surprise is a proliferation of certifications and it leads us to us the question how many certifications are enough, of course we ask first of all how many certifications are there well, there's a report from credential engine that came out in 2019 focused only on the United States and it says it has at least 738,428.
Unique credentials and there are there, you know, there are badges that are produced by post-secondary or credentials produced by so post-secondary educational institutions credentials produced by MOOCs, non-academic organizations secondary schools, so on and so forth. Tons of these credentials. There are a number of credential organizations as well. This is a website from credential finder.
Credential finder lists 70 distinct credential organizations. They probably could list many many more distinct credential organizations. And it raises the question of credential transparency as well. So there's a thing that's been created called the credential transparency description language or CTDL, which is a dictionary of terms and I'm quoting from it that provides a common approach to describing all types and levels of credentials from diplomas badges certificates, etc.
So you see all the stuff here that's involved. In creating this credential infrastructure all of the information about the person all of the information about the the Institution the badge information the evidence for the badge and so on. It goes without saying I think that having all of this information out in the open internet and here I'm looking forward a bit is a bit of a privacy nightmare because you've got personal addresses you've got specific information, but this is just the top level of the privacy nightmare that we might be looking at.
So credential applications, you know, there's a number of different organizations looking at how credentials can be used Academy for sorry Academy one for example is looking at transferability of credentials back in the days of metadata, we would talk about crosswalking from one metadata specification to another. This is kind of like crosswalking but for, Credentials specifically as opposed to credential formats, although if we have a number of different credential standardization languages and gee what are the odds then we'll have that issue as well LRNG uses the credentials to help you access learning opportunities from both local and national providers and then US military and and for that matter the Canadian military and, And other militaries are working with advanced distributed learning in something called competencies and skill systems, which again is a large credentialing program.
Okay, so we've got seven hundred thousand some odd certifications, how do we evaluate these things there's no easy way to answer that question because you know, what are the standards for evaluation of credential certification agencies, we have the accrediting boards in in Canada and the United States and elsewhere but they're not going to be looking at individual MOOCs, they're not going to be looking at credentials issued by companies, etc there's a company out there called credential of.
Valuation services, in fact there's a organization out there called the alliance of credential evaluation services of Canada, so now we've got multiple credential evaluation services about evaluating multiple many times multiple credential issuing agencies, etc.
And then there's the sign there's no sign that this is going to slow this is you know, from what we can see anyways it's only going to increase one person said recently on hearing about Microsoft's GitHub certified practitioner exam he commented this is Microsoft grifting, you know, you know, everybody is getting into the business of issuing credentials, everybody's getting into the business of evaluating.
In credentials Walmart, for example is partnered with guild education and others to provide credentials jet blue has a scholars program to evaluate credentials of employees. I could go on there's a huge list of these things another issue related to the existing credential system is exactly who's interest is being represented in the credentialing system, you know, a employers.
Tend typically to issue credentials that are specific to themselves so credential issued by one employer as as the article from an article from forgotten his first name. Kerzweil on the 74 says it may not be portable to another employer employer issued credentials to run the risk of being scaled back or disappeared when companies make cutbacks or when company.
Is merge or when companies go under but even more to the point there's the risk that credentials issued by employers encourage people to pursue their own the companies short-term interests rather than the employees long-term interest so as we push toward, you know, a competency-based badge-based credential-based system where we've got this infrastructure of credentials we have.
To. Ask you know, who is managing the system for who's benefit is this being done? And before I raise the issue of credential transfer, this is literally going to be amazed. It's already amazed in the academic world. We have an entire science of articulation agreements from one institution to the other of course, no institution wants to bring in too many credits from another institution, no matter how equivalent they are.
And with the multiple set of credentials this is going to only increase. Now we have organizations like the American College of Educators doing things like evaluating whether this particular certification produced by this particular MOOC say is equivalent to so many hours in a course of this nature say but the ACE assessments nonetheless are purely voluntary on the part of institutions, so you have to have the assessment.
Then the agreement body institution not the assessment was okay. And finally, and maybe not finally but you know things can get I'll say bad and weird but I don't don't really like that kind of language for this but still in my notes all I say is hmm, you know, we have.
Credentials and assessment for children. Here's something called child folio more recapturing photos videos notes tagging the child's interests assessing the child, you know, we have here in it's hard to read but attention maintenance self-comforting imitation, so these are for a really young children and you know, is it really appropriate to start issuing badges for self?
-control of feelings and behavior to a kindergarten student is all I can say about that. It just doesn't seem right to me. Okay, pause. Take a breath. The talk is about open recognition networks and not open credential networks for a specific reason and the reason is that credentials badger certificates whatever are intended to be forms of recognition right but recognition isn't just you know, giving people tests etc.
Recognition is actually a very broad concept and I'm gonna broad it almost. Writing it almost beyond reason here. We see recognition used all over the place. Here's a place onboarding in companies. Recognizing that somebody has joined the company providing them assistance proactive detection of you know, maybe bad practices whatever things like giving them a well-compacted first-day card, you have to actually discover and in large organizations, this is a non-trivial thing after discover that a new person.
Has joined the company or the organization. You know when I joined NRC and 19 years ago, it was kind of like NRC didn't notice that I joined I didn't get a welcome package or anything like that. So it's non-trivial but also recognition involves things like pairing new employees with recognition ambassadors, how do you do that or having new employees experiment with different kinds of ways of recognizing?
What they've done. Recognition also happens in career management. We talked about certifications and courses and stuff like that, but also to the work experience that they do seeing if they're interested in or qualified for a vacant position giving them certification for the work that they've done on the job these are elements of recognition in the workplace.
Recognition goes beyond merely employee related. Stuff there might be for example recognition of diversity and inclusion in an organization in a branch of an organization all the way down to looking at how a specific person practices diversity and inclusion in their own behavior in the workplace, you know, do they show efforts to help and guide other people who are less experienced?
Do they you know, who do they seek out for approval? These are. Sorts of things. Do they treat people with disabilities equally to the way they treat other people? Again, these are recognition kind of tasks, although you wouldn't give a badge right to somebody for being nice to a person with a disability.
That would be kind of ridiculous wouldn't it but it's part of the overall picture. And then of course performance and goals the regular sort of stuff that we would want to recognize both in the workplace and I in in school as well or in a university classroom. You know both you know, there's two sides of it here in this diagram from Buck Manning University of Victoria engagement on the one side and performance on the other side and we could go all the way through to the.
Um, The the different levels of recognition, you know recognition of what they've done how they felt about what they've done what the impact of what they've done has been and what the overall result has been toward the organizational schools, you're matching it up with the Kirkpatrick scale, of course evaluation say, But there's more to recognition than that there's pure recognition and this is a recognition now that is you know goes beyond the workplace certainly there's pure recognition in the workplace there's peer recognition anything like academic networks in the very nice introduction that I got for this talk there was an element of peer recognition when they said, you know, they're very pleased to have me that's a way of recognizing, you know, we're very glad to have them now the right ways to do this and there are wrong ways to do this here's The wrong way close.
You might remember not lamented organization called clout to what they did is they created something called a close score and what they did is they gathered statistics across nine different social networks Twitter Facebook etc etc and according to them thirty six hundred features that quote capture signals of influential and interactions and aggregated across multiple dimensions for each user and so you get this.
Completely opaque close score and really, you know, and you look at the dashboard you'd see Twitter 500 plus mentions you're on 200 plus list you get retweets etc now. I think we all know certainly at this level that counting retweets and counting followers is not really going to be the basis for a good recognition network remember we're in that circle now.
Awarding the badge recognizing the achievement, how are we picking out how to recognize someone in the workplace? You know, there's the different elements of recognition that we are told result in different positive or better or negative benefits, you know, the recognition in the workplace may improve happiness values humanity of the workplace etc, but of course if you did that using a cloud-based system or if you did that you're saying they used to have these things called attaboys, you probably wouldn't get these benefits the way you gather data.
A you recognize somebody really has an impact on the benefits you're going to get from recognition we had these directors awards right in the directors rewards, we're completely arbitrary for no purpose whatsoever except to recognize friends that would have a negative impact in the workplace. Sony how this is morphed over time into a whole field of employee recognition technology everything from bucket list which has some things for employee recognition to appreciate something called kudos something called gusto with to use but you know, look, I don't know if you can see the slide but if you look closely at this example from bucket list, you've got some really wackadoo associations and you.
Use wackadoo in a very precise and tacky technical sense, for example, one item is on the list get married, okay, that's a bucket list thing for people why is it on this person's list based on interest food? Ah or another one learn to illustrate based on location. North Vancouver.
Or this one. I liked eat pasta in rum sure that's on a lot of people's bucket lists based on interest nature, you know. We can go really wrong in these kind of recognition things and and if employee recognition is is a soundly based is these recogn recommendations then it's gonna have problems so we have recognition and social media as well, you probably all have Facebook page.
I'm sure you have Facebook page pages that maybe some of you don't so Facebook not Facebook. LinkedIn I should say, This is an example from my own LinkedIn page. I don't use Facebook. I don't know why I said Facebook anyhow so here's me and it has you know for science endorsed by Pedro Fernandez who is highly skilled at this so that's good and four of my colleagues out of an organization of 1700 research scientists, but also some other industry knowledge in here again are just scores and if you look at it, it's just the number of people who have recommended me.
At in this. Not good none of this is good, right? So pausing again. We have the whole competency thing. I have the whole wrecking mission thing but so far to my estimation it's a house of cards right it's not really based on anything substantial and it's gonna come back to how we're recognizing these but recognizing in a different sense and we've been using it so far.
And the sense that I'm thinking about is. The epistemological sense the idea that recognizing is a way of knowing things let's think about this.
Think about how you know somebody or how you know, it's a person that you're greeting at the train station you look at the people getting off the train and there's a bunch of people there this platform isn't very crowded which helps but you see somebody and you recognize oh yes, that's my grandmother.
How are you doing this well it's not her clothes score that's for sure it's nothing even remotely resembling anything like what we've done so far it's more like where's Waldo where once we see Waldo we go, ah there's Waldo and it's an immediate recognition kind of thing now how is this happen how does this happen in humans how does this happen in technology well recognition is something that can be described using neural networks?
Here's a a nice picture from Steve Easterbrook, it's fairly characteristic you have the one in input say a image broken into pixels and then your neural network is going to do some feature analysis detect edges combinations etc and come out with an output dog, so this neural networks very specifically recognizes a picture of a dog.
Neural networks also are useful because they can learn one example of the way neural networks learn is something called back propagation and the way that works is basically you already know what a dog is and so you give the neural network an image of a dog you feed the signals through and if it comes out not dog, you send the correction back through the system and that's what the red arrows in this diagram represent is the correction.
That's being sent back and what that correction will do is it will adjust the weights of the nodes in the hidden layer that's the w on the on the specific on the little circles there and it will also adjust the strength of the connection or the weight of the connection between one node and the other and in newer neural network algorithms, it can affect a bunch of different parameters, for example decay rates, for example.
For firing things like that and the idea is you tweak these parameters enough based on this feedback and and. Eventually your neural network after a certain number of rounds of training will reliably recognize a dog and we know that works we know it works well enough that we've got a variety of neural network recognition tasks such as classification clustering regression, which is the detection of statistical trends dimensionality reduction and more.
So this is how neural networks work and to a large degree this is how human recognition works as well, if you look at the work of people like David Marr and others unsee the visual cortex you see those layers of neurons you see the edge detection layer you see the two and a half dimensional image layer going back through and then on into the deeper levels of the brain, so at a very simple level we know recognition is a neural network phenomenon in human.
's and probably a deeper levels as well but it's not foolproof none of this is foolproof there are all kinds of ways for things to go wrong with recognition optical illusions and here's one right the the classic these lines are straight really no really kind of illusion there's also things like confirmation bias there's also equivocation we might see the images as a duck who we might see the images as a rabbit, it's really.
Hard for us to see new things it's really hard for us to detect errors in our own recognition. So artificial intelligence is being used the recognition capacity as being used to evaluate performance in different domains and disciplines and I could do the rest I could do this for the rest of the day but I don't have the rest of the day but here's a few ballet artificial intelligence evaluations of ballet best worst, you know, where's the shin where's the thigh etc?
Here's another one surgery and it's interesting you can do this yourself if you look at these sutures from left to right which ones are the good sutures that's pretty easy to tell isn't it and we don't need to be experts even remotely in surgery to be able to tell well probably the one to the right is the good one the one to the left here has horrible, you know, the lines don't match up there's I mean, you know, I would certainly.
Go out. The surgeon on the right to stitch me up. Here's another completely unrelated feel construction and you know, the image here makes it kind of clear for us how well is the construction task going? We see you know how much of it they've completed of this particular task, we can see are the bars the rebar is it being spaced properly how many people are doing the work etc all the way down to you know, are they.
Following safety protocols, do they all have their helmets on I'm not sure about this guy here. Oh yeah, okay. I think he has his helmet on there all right and one that I thought was a lot of fun especially for me how normal are you and this is a case of AI recognition looking at my image and I give it permission to access my webcam for this for the rest of my life, no I'm kidding and it's at how normal is this person based on this image and the answer was.
Pretty normal which yes surprised me too, it also said I'm gonna live to be 84 so better would be nice but you know considering the way I've lived my life. I'll take it so okay. So that leads us to the whole concept of recognition networks generally we've got this concept we've got this technology, where are we going with this well?
We've got activity recognition different technologies actually doing the task and here we have gaze point data object recognition action recognition, etc, we have systems that have been developed to do this now this is something that was created by Hitachi well known company and AI for human activity recognition if you look the human activity recognition up on Google there's a whole pile.
Of stuff different methods have been defined for activity recognition gestures events behaviors group actions, and so on and and on the left-hand side we see the different kind of activities diving playing golf running. I have a an application called Strava on my phone with tracks all of my running or not running.
I don't run let's not be silly all of my hiking and cycling activity. Technologies that are being used for these recognition networks include wearable sensors integration devices, which is where my strava comes in which uses a GPS communication systems the internet bluetooth etc and then the whole storage and integration in the back and all of this is important because we don't think about it.
And then the applications for this kind of recognition thinks like creating new initiatives setting up new kinds of gig employment focusing on development things like that, we at NRC created something called micro missions with the Treasury Board Secretariat where we used AI to match individual employees with individual micro missions or small learning options.
Now, there are challenges to this right and and here's a list from a paper and research gate well not in research gate, which you know what I mean? Recognizing parallel activities like cooking and speaking on the phone overlapping activities vagueness in recognizing activities and recognizing activities when there are multiple occupants in a room.
And there are ethical challenges to this whole thing. You know, there are challenges in human activity recognition. For example, invisibility recognizing people that AI doesn't even recognize exist finding the people maybe who are poor who do not have internet active internet access or even systems that look at only certain groups of people there's.
Complaints, for example about Instagram and tick tock using only beautiful models and ignoring the rest. There's the problem of bias in these recognition networks Amazon engineers, for example spent years working on a hiring software but gave up because they could not find a model that doesn't discriminate against women.
That's a problem. Also, the question of explainability and interpretability of these recognition engines. You know interpretability does it actually get the cause of the effect correct as opposed to you know, a meaningless association or correlation and then explainability can we actually talk about why it gave the result that it didn't and you can see the problem in this diagram here the, Explanation or the AI methods that are more explainable are significantly less reliable and the methods that are more reliable are significantly less explainable and that's going to be because our explanations are in abstractions.
They're in sentences words models etc which miss out a lot of the finer detail and the the deep learning neural nets are sub symbolic they're capturing things. That. We don't even have words for which helps them be accurate, but then how do you explain how do you even tell whether they've predicted correctly and based it on well ethical factors?
And then of course there's the whole issue of surveillance culture and surveillance culture is not just you know cameras in the public square surveillance culture has everything to do with us sharing working in social networks etc which brings us back to open recognition networks and this is I think where we're going to go instead of badges and blockchains, etc we have all of this infrastructure on the one side right we've got all of the, Infrastructure for AI task detection performance evaluation etc, but we have big problems with ownership and control now the way this began in the online world in the academic world was with e portfolios and there are different ways of doing e-portfolios and different outcomes of e-portfolios diagram here is kind of a Google 100% Google solution right where you're using Google notebook and you're using.
YouTube and Google Docs and all of your stuff is stored in Google apps, but of course a lot of learning management systems stop doing that you silly thing a lot of learning management systems did this all in-house in the whole competencies and skills systems thing that I was talking about earlier, they have activity records using something called the experience API these are based on automatically gathered and stored.
Recordings of your activities, but the thing is who owns that learning records store probably not you and that's what brings us to the question of ownership and choice in recognition networks there are all of these artifacts that we're creating that we're going to create both as our content creation as a result of our activities, etc.
And we can collect these select them reflect on them and then create different portfolios or different ways of providing assessments but only if we own this in other words only if it's a distributed kind of system and not a centralized kind of system, so there are different platforms that are available the centralized system is going to look like Facebook it's going to look like Twitter and it's going to end up having an impact a lot like, Out where it's using these broad centralized measures based on aggregating numbers of things but if we want recognition networks to be more narrowly focused focused on specific occupations specific groups of people and better preserving our privacy we're probably looking more at federated networks or possibly even independent networks there's a question there of whether the independent, you know, each person.
Has their own system will provide enough interactivity to make you know, performance recognition possible there are applications that are allowing this including one called freedom bone, which is a home server that you run in your house that has all of these records for you stored locally and then you share them where you want to.
This is only the beginning there are numerous challenges ahead potential social ethical and cultural challenges, how do we define what we share what kind of network structure is appropriate for an overall system of AI recognition that is explainable and and correctly attributes cause to effect what is the ownership of this you know, how do we handle the ethics how do we make sure that everybody?
Is evaluated on a fair and equitable basis and then just how do we put all the pieces into place all of these are issues? And we have. 45 seconds for discussion. I'm very sorry about that, but I hope you found this interesting.
Oh absolutely yeah, thank you do you have Steven do you have time for just a brief chat here we have a comment and interesting comment from Nate Auto
I have time it's up to you guys and the next speaker most importantly.
Of course Nate would you like to just jump on the microphone and make your comments? Sure and soon thank you so much for the presentation and for the good thinking around this. I mostly agree with one of the points that you made sort of at the end which is that we need to think about on what data is open is like the calculation of value occurring and if we aren't intentional about creating data sources that have the type of equity of access and and good information about people in the ability for all the forms of recognition that our community can put into that pool if we don't.
Build. In your data pool that has those characteristics we're going to get some pretty unequal outcomes out of it when value is calculated about what skills people have or whose suitable for what jobs are or other opportunities
yeah and so if we don't start turning to delivering that kind of value based on the forms of open recognition that we're building then we'll be stuck in a world in which some company builds an algorithm that is based on different data, yeah and and I mean, they're they're gonna be elements to that as well, we typically.
Talk as others gonna be one great big pool of data there probably won't be and we need to talk about how to organize a different pools of data what the sources for those pools of data will be we you know equitability very important is just one of these things what counts you know, as an expert performance when we're using our data to train these systems, you know, if we're if we're recognizing doctors, we want to use expert doctors.
As the training data and not bad doctors but what counts as an expert doctor certainly the question of whether or male or female is irrelevant to this calculation either a man or a woman can be an expert doctor, but what are the other kinds of properties in a person that do const constitute expertise in a doctor and how do we select those doctors and that's going to be our job the job of humans in the future is to be defining what we think is expertise and picking these examples.
So the social network aspect of this is going to be an integral part of it.
We also had a question here from from Barrett tool bear. D would you like to ask your question? Yes just wondering on your comments on how recognition is not well captured currently on the web to what extent would you say that this is a problem of separating reputation from recognition how do they those concepts relates in the digital world currently?
Yeah. I mean reputation basically is data that originates from a person whereas recognition is data that originates from some sort of evidence that's a broad, you know, and then probably vastly over caricatured characterization but that's what gets to the heart of the problem right and data from individual persons.
Isn't reliable from a for so many reasons like you know, the whole eyewitness effects where. People say they saw something but they did not could not have etc to bad actors gaining the system etc so you need to tie a reputation system to a recognition system otherwise it doesn't work.
I think this is one of the reasons why the whole concept of web of trust kind of failed it was just too easy to gain and and and why similar kinds of systems based on opinions rather than evidence here. I'm thinking of prediction markets are also easy to. Gain, so that would be my first kind of pass at a response to that.
But it's a deeper issue for sure.
