– An interview with George Siemens
Interviewer: Wei Xuefeng
Prof. stands for Prof. George Siemens, and Interviewer stands for Dr. Xuefeng Wei.
Prof：Hello.
Interviewer: Hello, Professor.
Prof: How are you doing today?
Interviewer: I’m very well. How are you?
Prof: I’m doing very well. Happy new year to you.
Interviewer: Yeah, I will, yeah, I can hear it.
Prof: You can’t …
Interviewer: Can you hear me?
Prof: I can hear you. Is my audio a problem?
Interviewer: No, no problem.
Prof: Ok.
Interviewer: Ok .Happy new year first.
Prof: Thank you. When do you have your new year?
Interviewer: Yesterday.
Prof：Yesterday, ok. All right, that’s right. Because it’s the second for you now, right?
Interviewer: Yeah, Yeah.
Prof: OK, right. Well, sorry about yesterday. I had listed in for summaries. And I didn’t have my laptop open so I didn’t get my reminder and I completely forgot. So, my apologies.
Interviewer: It doesn’t matter.
Prof: Ok.
Interviewer：You are very welcome.
Prof: Yet, well, Ah…So this is an IEEE event, an IEEE publication. And we understand that you just want to talk a bit about “learning analytics”. Is that correct?
Interviewer: Yeah.
Prof: Ok. Well, I let you start with however you want to proceed with your questions and we can go from there.
Interviewer: Ok. The list of the interview questions I have sent to you. Have you received it? Or now I will…
Prof：We just quickly pull up here because you sent that to me a while ago , right?
Interviewer: Yeah, I will send to you now?
Prof :Ok. Oh, here I go, I’ve got it here. You sent to me a list on November 9th.
Interviewer: Yeah.
Prof: Yes. So I have that list in front of me now.
Interviewer: Ok. The first question is about “What is learning analytics? And why learning analytics emerged? And where learning analytics is being used?”
Prof: Ok. Now, what’s the easiest way for me to do this? Did you want me to type in and answer to you? Or do you want me just to respond to the question now? So do you want the audio answer or the text answer?
Interviewer: You can speak and I will transcribe it.
Prof: Ok. So it’s up to you, I mean, if you want. Ah…If you think that and I can answer these questions now and you can decide whether…Look! Are you recording all these or are you using these to transcribe to put on the website?
Interviewer: Yeah, yeah. Professor Kinshuk wants me record our interview. I’m recording now.
Prof: Ok, good. Let me just kill that background noise. I will be back in one second. And we can answer it. Ok, well, I’ll start with the first question then, you are looking at, you know, what learning analytics are, and how and why they emerged, and where they’re being used? A quick definition the way that we’ve described through the Society for Learning Analytics Research is that learning analytics is essentially the process where we measure, we collect data that is actively generated through the students’ participation in a particular learning activity. In that process of collecting data, we can gain insight into what students are doing, how they are developing, what sort of their next stage of activity might be. It’s essentially the measurement collection of this data that students are generating. Then we use that to report back both the learner and the institution and the educator as well – we report back to at least those three agents. And our goal is really to understand better how is learning happening and how can we improve the learning experience or the environment in which learning occurs. So, that’s our definition. If you want something a little bit shorter, basically it’s analyzing and communicating the patterns in the data trails that students generate as they learn.
Now. why learning analytics have emerged? I think it’s due to quite a few reasons. Probably one of the biggest ones is just the amount of data that’s now available. There’s a significant amount of data available that we are collecting on students but we didn’t have in the past. Now we collect data about the students’ experiences in the educational process that in the past wasn’t available to us. We can collect data on students, how often students use the video, if they are in a learning management system, what kind of activities they are engaged in, how much time they spend on test. We can also start to create learner models based on the activities that will let us know which student might be at risk of dropping out.
We can also look at ways to personalize the learning experience by evaluating what students already know and how we can help them with what they need to know next. So, there is a variety of things that have come out of it. Most of those have emerged because students now leave these data trails. It also corresponds to some of the activity that has happened in broader landscape of society. You know, business intelligence is a big factor now. A lot of businesses are investing heavily into using or analyzing the data that they’re collecting and using that to better run their organization. I think in that regard it’s connected or related to the educational landscape we are talking about here.
Your third sub-question is where learning analytics now being used? It’s really quite broad, they are being used in numerous settings. Definitely from a university, for instance, there are a variety of systems where significant amount of efforts have been devoted into system level tracking to be able to understand what’s happening with students, how are students succeeding in the different contexts. We’re looking at what’s the outcome of student activity as they connect socially, the networks that they produce in their interaction process. It’s being used to create, you know, the students’ profiles or models and, as I mentioned earlier, early warning systems are also being used to train and create a more personalized learning experience. So rather than students taking all of the courses that have the same content in the course, the system starts to know what the student already knows and begins to personalize that experience for each individual student. Much of the analytics activity going on has been borrowed from other fields, and some of it is emerging. The landscape certainly is quite broad and developing. The most innovative activities are coming from companies right now. Unfortunately, universities-higher education – has been little bit more delayed in adopting learning analytics. Well, many for-profit universities have been very aggressively moving forward with utilization of analytics to improve students’ experiences.
Interviewer: OK. You said about the LA that it refers to the data mining, about the student learning. So the next question is: what are the differences between Learning Analytics and Education Data Mining?
Prof: It’s worth emphasizing the overlap in EDM and LA. Numerous conversations in the educational data mining relate to activity in learning analytics. We had significant overlap between EDM and LA at our conference in Vancouver this last year. We invited a group of representatives from IEDMS to present to the Learning Analytics Conference because we want to make sure that we are working together wherever it makes sense for us to collaborate.
I did a paper with our Ryan Baker who is head of the International Educational Data Mining Society for the proceeding for that conference where we looked at the distinctions between these fields and particularly, our goals to move forward with more collaboration. We want to share research ideas so that we have better quality of research and friendly competition. We engage in with each other. Through research and cooperating, we feel we will improve the quality of our respective societies.
A few areas where we are going to see some of the more significant distinctions in educational data mining and it comes together with the paper I did with Baker, but educational data mining society really emphasizes automated discovery, so the emphasis is heavily on models and discovery of patterns. With learning analytics, there is still human judgment involved, even though we rely on automated discovery as well. Learning involves social processes. Holistic social processes, such as teaching and learning, there is greater emphasis on interaction and intangibles that may not be captured with today’s machine learning models. We want to capture the broad scope of learning, the whole experience, the complex experience. Educational data mining is more focused on individual components and relationships between them. EDM does not necessarily take a holistic approach. And with learning analytics, we also look at the contexts as well as the particular interventions. Our emphasis is quite strongly on trying to make sure that we have a broad integrated overview.
Another difference relates to LA and EDM origins. EDM does have an origin in the intelligent tutoring field, student modeling and educational software and there is more focus on predicting outcomes of students and there is a lot more talk on predictive modeling. Within learning analytics, these elements are also included, but there is greater emphasis on systematic interventions and increased focus on personalization and adaptivity. LA is also concerned with semantic web and linked data, to be able to target the needs of students’ base of the knowledge by profiling what they know in relation to a particular domain of knowledge.
Probably the best way to look at the distinctions between EDM and LA is to look at the tools and the techniques. EDM relies heavily on clustering, where they are looking at understanding relationships between entities at a granular level. They also rely heavily on intelligent tutors and place greater emphasis on creating models. If you look at the learning analytics side, there is greater emphasis on social interventions. Social network analysis, for example, is commonly used in learning analytics. Discourse analysis is also common, looking at the sense making process so there is an analysis of discourse that considers rhetorical moves as individuals engage with each other around a topic in order to better understand it. It is important to emphasize that there are many areas that overlap between EDM and LA and that going forward some of those areas will continue to merge.
Interviewer: I can hear you, but I can’t see the video.
Prof: OK, do you want to go on with the next question or.. What’s your interest?
Interviewer: OK. While the context is very important, so how can one analyze the context of learning?
Prof: Well there are various ways but I think it’s still at the early stage. Context in learning analytics models currently relies heavily on inputs that the students have into a particular system, such as a learning management system or students information system. And this works now, but it really isn’t the best approach going forward. We need something more diverse in data collection. For example, if you look at some of the developments with Xbox 360 or if you look at other systems that are sensor-based, it can provide a reasonable quality of context input for gamers. For LA, we need to think about how to better capture environmental & contextual factors. Even if context determination is limited where it provides basic resources about students’ profiles such as log in times, mobile devices, or, if in a work setting, location and information needed to perform a particular type of work, it can still provide important benefits to both the learner and the system.
But broadly speaking, right now context is captured through the students’ use of technology, through their browser, through information that’s collected possibly from apps and in some cases, from their mobile devices. Going forward, systems will be developed for capturing the contexts through wearable computing and other context aware devices. Currently, we don’t have enough of data collection processes in place yet. We are relying on too heavily on keystroke, keyboard and browser data and we are not tracking the broad scope of the learning context.
Interviewer: ok. Context is very important, LA pays more attention to the learning process. How can one analyze learners’ learning process through the Learning Analytics?
Prof: Sure, it’s important question. The process of learning is critical and we want to be able to capture as much of that as we can. By collecting better quality data, we are able to give them feedback and more specific directions when needed. We can also use learning analytics to generate student models and profiles so that we can automate help and support. The specific questions that we want to answer regarding a student’s learning process will vary based on the knowledge discipline, technologies used, and related factors.
One another approach presented by Abelardo Pardo during our LA conference in Vancouver in 2012, is particularly effective at capturing rich in-context data. He did a presentation about the use of virtual machines to capture data about the students programming habits. The students would do all of their work on programming activities in the VM, which allowed the researcher to capture everything they were doing and each point of learning. So if we want to understand learning process, that would be my thinking: either we create student models, based on high observation settings that can then be applied to another settings or in systems by which understand more and more the students processes, such as all that were mentioned by Abelardo Pardo.
Interviewer: you mean, the process, the learning process pays more attention to the cognitive process?
Prof: Understanding the learning process requires watching students engage with content and each other. There are two ways of doing that, one, as I mentioned, you can create settings where you capture great amounts of data on students. This can occur through automated collection of student data such as the VM example provided earlier. The second approach involves manual data collection, in addition to automated processes. This approach is high cost as it involves high observation settings such as lab environments where data is manually entered. This is not unique to educational data collection. Google Maps, for example, requires significant human manipulation of data to ensure quality experiences for end users. It is unrealistic to expect that quality LA can be conducted without the addition of observational and faculty contributed data.
Interviewer: yes, ok. So the LA analyzes the data, but how do we deal with privacy and data protection in learning analytics? Are there any best practices one can follow?
Prof: These questions are coming up more frequently because people are becoming concerned about the privacy and data ownership. Unfortunately, best practices have not been defined. There has been a fair amount of dialogue, but privacy laws vary by region. IT departments obviously have standards and benchmarks for secure storage of data. Guidelines for use of that data for analytics are less established and vary from institution to institution. Another challenge concerns the nature of learning today where learners use software tools and different online resources. Each software provider will have different standards regarding data access. Protecting student’s privacy when a range of university-hosted services and third party cloud-based services are used becomes a significant challenge. In the future, we may end up viewing privacy as a transactional entity, where learners indicate their willingness to have universities analyze their data in exchange for better support services and more personalized content.
Interviewer: some people might worry about the effectiveness of LA. So what are the methods for evaluating the effectiveness of LA?
Prof: Evaluating the effectiveness of LA is a function of the goals of a universities deployment of analytics. Currently, much of the language around LA relates to student success and preventing dropouts. If this is a target for universities, the effectiveness of LA is determined by improvements in completion rates. Other indications of success might be in helping to reduce degree completion time, reducing costs by providing more focused support to learners, and so on. Ultimately, LA should be about optimizing and improving learning for all learners, not only about reducing costs or preventing dropouts.
Interviewer: Ok, the future of LA is very prosperous. What are the limitations of LA and what is the future of LA in your opinion?
Prof: There are limitations with learning analytics, as is the case with any single approach or metric used with the intention of improving education. The biggest right now is the scope of data capture. So much of our data right now is being captured from learning management systems, keystroke data, browser data – basically what people do through computer. There are many learning experiences that are, most actually, real world contexts. We don’t capture enough of those experiences yet. The biggest challenge, I would say, we face with learning analytics, its broadening the scope of data capture so that we are actually able to provide more insight into learning and teaching processes.
The other limitations right now are that the social aspects of learning are difficult to reduce to an algorithm. Currently, we might look at student logins, how much time they spent in a course, if they accessed readings, which videos they watched, and so on. These give us a bit of insight but there many other things that we need to capture to understand the learning process.
In response to the second question about the future of learning analytics, there are two important aspects. One relates to real world data collection that accounts for the context and environment involved in learning. More data points need to be captured regarding context. The second aspect is mobile learning. Mobiles offer numerous data points about student movement, information seeking behavior, social interactions, environmental factors such as location, etc. Mobiles and wearable computing (such as Google Glass) are going to be an important part of data collection in the future.
Interviewer: OK, LA has a prosperous future.
Prof: Yes, I think the future of LA is positive from research perspective. I think you will see significant research in this area going forward. There is interest now developing by grant funding agencies. We are also planning a summer analytics institute next July at Stanford University where we will try to bring together thinkers and theorists from different fields. We are not just looking at these clusters of educational technologists, statisticians and programmers. We really want to start bringing in cognitive scientists, learning scientists and so on. We already have focus on computer supported collaborative learning as some of the learning sciences represented. But what we really need is broadening the scope of that representation, going forward. Once grant funding agency will start to create funding programs that help schools or university researchers take advantage of the potential learning analytics, the field will have crossed an important threshold.
Interviewer: yes, in china, that’s some researchers now do some research about LA, so in the future, we can cooperate in the area.
Prof: that would be great.
Interviewer: so in the few days, I will try to transcribe the interview and send it to you for checking it. Can you send some text materials about LA to me?
Prof: ok. I will try to find a few resources that you may find useful.
Interviewer: yes, yes, ok. In last March, you give us a course, an international course about connectivism when I was studying in Beijing Normal University. Do you remember?
Prof: I do, yes.
Interviewer: My research is about the “Cognitive Process Simulation of Problem Solving in Primary Mathematics and its Application”, cognitive process analysis and simulation are important parts in Learning analysis. I graduated last summer and now I am working at Ludong University in Shandong province, Yantai city, a beach city. So when you come to China, welcome to visit our university.
Prof: OK. It sounds great.
Interviewer: OK. Thank you for your interview, professor.
Prof: All right. Thank you, it’s good to talk.
Interviewer: OK. Thank you.
This interview has been published in Journal of China Educational Technology, 2013（9）：1-4.（CSSSI）.
To cite this article:
WEI Xuefeng & SONG Lingqing (2013). Understanding the individual students better with learning analytics [J]. China Educational Technology, (9):1-4.