research papers in learning

• Tutorial Review
• Open access
• Published: 24 January 2018

Teaching the science of learning

• Yana Weinstein   ORCID: orcid.org/0000-0002-5144-968X 1 ,
• Christopher R. Madan 2 , 3 &
• Megan A. Sumeracki 4  

Cognitive Research: Principles and Implications volume  3 , Article number:  2 ( 2018 ) Cite this article

273k Accesses

101 Citations

757 Altmetric

Metrics details

The science of learning has made a considerable contribution to our understanding of effective teaching and learning strategies. However, few instructors outside of the field are privy to this research. In this tutorial review, we focus on six specific cognitive strategies that have received robust support from decades of research: spaced practice, interleaving, retrieval practice, elaboration, concrete examples, and dual coding. We describe the basic research behind each strategy and relevant applied research, present examples of existing and suggested implementation, and make recommendations for further research that would broaden the reach of these strategies.

Significance

Education does not currently adhere to the medical model of evidence-based practice (Roediger, 2013 ). However, over the past few decades, our field has made significant advances in applying cognitive processes to education. From this work, specific recommendations can be made for students to maximize their learning efficiency (Dunlosky, Rawson, Marsh, Nathan, & Willingham, 2013 ; Roediger, Finn, & Weinstein, 2012 ). In particular, a review published 10 years ago identified a limited number of study techniques that have received solid evidence from multiple replications testing their effectiveness in and out of the classroom (Pashler et al., 2007 ). A recent textbook analysis (Pomerance, Greenberg, & Walsh, 2016 ) took the six key learning strategies from this report by Pashler and colleagues, and found that very few teacher-training textbooks cover any of these six principles – and none cover them all, suggesting that these strategies are not systematically making their way into the classroom. This is the case in spite of multiple recent academic (e.g., Dunlosky et al., 2013 ) and general audience (e.g., Dunlosky, 2013 ) publications about these strategies. In this tutorial review, we present the basic science behind each of these six key principles, along with more recent research on their effectiveness in live classrooms, and suggest ideas for pedagogical implementation. The target audience of this review is (a) educators who might be interested in integrating the strategies into their teaching practice, (b) science of learning researchers who are looking for open questions to help determine future research priorities, and (c) researchers in other subfields who are interested in the ways that principles from cognitive psychology have been applied to education.

While the typical teacher may not be exposed to this research during teacher training, a small cohort of teachers intensely interested in cognitive psychology has recently emerged. These teachers are mainly based in the UK, and, anecdotally (e.g., Dennis (2016), personal communication), appear to have taken an interest in the science of learning after reading Make it Stick (Brown, Roediger, & McDaniel, 2014 ; see Clark ( 2016 ) for an enthusiastic review of this book on a teacher’s blog, and “Learning Scientists” ( 2016c ) for a collection). In addition, a grassroots teacher movement has led to the creation of “researchED” – a series of conferences on evidence-based education (researchED, 2013 ). The teachers who form part of this network frequently discuss cognitive psychology techniques and their applications to education on social media (mainly Twitter; e.g., Fordham, 2016 ; Penfound, 2016 ) and on their blogs, such as Evidence Into Practice ( https://evidenceintopractice.wordpress.com/ ), My Learning Journey ( http://reflectionsofmyteaching.blogspot.com/ ), and The Effortful Educator ( https://theeffortfuleducator.com/ ). In general, the teachers who write about these issues pay careful attention to the relevant literature, often citing some of the work described in this review.

These informal writings, while allowing teachers to explore their approach to teaching practice (Luehmann, 2008 ), give us a unique window into the application of the science of learning to the classroom. By examining these blogs, we can not only observe how basic cognitive research is being applied in the classroom by teachers who are reading it, but also how it is being misapplied, and what questions teachers may be posing that have gone unaddressed in the scientific literature. Throughout this review, we illustrate each strategy with examples of how it can be implemented (see Table  1 and Figs.  1 , 2 , 3 , 4 , 5 , 6 and 7 ), as well as with relevant teacher blog posts that reflect on its application, and draw upon this work to pin-point fruitful avenues for further basic and applied research.

Spaced practice schedule for one week. This schedule is designed to represent a typical timetable of a high-school student. The schedule includes four one-hour study sessions, one longer study session on the weekend, and one rest day. Notice that each subject is studied one day after it is covered in school, to create spacing between classes and study sessions. Copyright note: this image was produced by the authors

a Blocked practice and interleaved practice with fraction problems. In the blocked version, students answer four multiplication problems consecutively. In the interleaved version, students answer a multiplication problem followed by a division problem and then an addition problem, before returning to multiplication. For an experiment with a similar setup, see Patel et al. ( 2016 ). Copyright note: this image was produced by the authors. b Illustration of interleaving and spacing. Each color represents a different homework topic. Interleaving involves alternating between topics, rather than blocking. Spacing involves distributing practice over time, rather than massing. Interleaving inherently involves spacing as other tasks naturally “fill” the spaces between interleaved sessions. Copyright note: this image was produced by the authors, adapted from Rohrer ( 2012 )

Concept map illustrating the process and resulting benefits of retrieval practice. Retrieval practice involves the process of withdrawing learned information from long-term memory into working memory, which requires effort. This produces direct benefits via the consolidation of learned information, making it easier to remember later and causing improvements in memory, transfer, and inferences. Retrieval practice also produces indirect benefits of feedback to students and teachers, which in turn can lead to more effective study and teaching practices, with a focus on information that was not accurately retrieved. Copyright note: this figure originally appeared in a blog post by the first and third authors ( http://www.learningscientists.org/blog/2016/4/1-1 )

Illustration of “how” and “why” questions (i.e., elaborative interrogation questions) students might ask while studying the physics of flight. To help figure out how physics explains flight, students might ask themselves the following questions: “How does a plane take off?”; “Why does a plane need an engine?”; “How does the upward force (lift) work?”; “Why do the wings have a curved upper surface and a flat lower surface?”; and “Why is there a downwash behind the wings?”. Copyright note: the image of the plane was downloaded from Pixabay.com and is free to use, modify, and share

Three examples of physics problems that would be categorized differently by novices and experts. The problems in ( a ) and ( c ) look similar on the surface, so novices would group them together into one category. Experts, however, will recognize that the problems in ( b ) and ( c ) both relate to the principle of energy conservation, and so will group those two problems into one category instead. Copyright note: the figure was produced by the authors, based on figures in Chi et al. ( 1981 )

Example of how to enhance learning through use of a visual example. Students might view this visual representation of neural communications with the words provided, or they could draw a similar visual representation themselves. Copyright note: this figure was produced by the authors

Example of word properties associated with visual, verbal, and motor coding for the word “SPOON”. A word can evoke multiple types of representation (“codes” in dual coding theory). Viewing a word will automatically evoke verbal representations related to its component letters and phonemes. Words representing objects (i.e., concrete nouns) will also evoke visual representations, including information about similar objects, component parts of the object, and information about where the object is typically found. In some cases, additional codes can also be evoked, such as motor-related properties of the represented object, where contextual information related to the object’s functional intention and manipulation action may also be processed automatically when reading the word. Copyright note: this figure was produced by the authors and is based on Aylwin ( 1990 ; Fig.  2 ) and Madan and Singhal ( 2012a , Fig.  3 )

Spaced practice

The benefits of spaced (or distributed) practice to learning are arguably one of the strongest contributions that cognitive psychology has made to education (Kang, 2016 ). The effect is simple: the same amount of repeated studying of the same information spaced out over time will lead to greater retention of that information in the long run, compared with repeated studying of the same information for the same amount of time in one study session. The benefits of distributed practice were first empirically demonstrated in the 19 th century. As part of his extensive investigation into his own memory, Ebbinghaus ( 1885/1913 ) found that when he spaced out repetitions across 3 days, he could almost halve the number of repetitions necessary to relearn a series of 12 syllables in one day (Chapter 8). He thus concluded that “a suitable distribution of [repetitions] over a space of time is decidedly more advantageous than the massing of them at a single time” (Section 34). For those who want to read more about Ebbinghaus’s contribution to memory research, Roediger ( 1985 ) provides an excellent summary.

Since then, hundreds of studies have examined spacing effects both in the laboratory and in the classroom (Kang, 2016 ). Spaced practice appears to be particularly useful at large retention intervals: in the meta-analysis by Cepeda, Pashler, Vul, Wixted, and Rohrer ( 2006 ), all studies with a retention interval longer than a month showed a clear benefit of distributed practice. The “new theory of disuse” (Bjork & Bjork, 1992 ) provides a helpful mechanistic explanation for the benefits of spacing to learning. This theory posits that memories have both retrieval strength and storage strength. Whereas retrieval strength is thought to measure the ease with which a memory can be recalled at a given moment, storage strength (which cannot be measured directly) represents the extent to which a memory is truly embedded in the mind. When studying is taking place, both retrieval strength and storage strength receive a boost. However, the extent to which storage strength is boosted depends upon retrieval strength, and the relationship is negative: the greater the current retrieval strength, the smaller the gains in storage strength. Thus, the information learned through “cramming” will be rapidly forgotten due to high retrieval strength and low storage strength (Bjork & Bjork, 2011 ), whereas spacing out learning increases storage strength by allowing retrieval strength to wane before restudy.

Teachers can introduce spacing to their students in two broad ways. One involves creating opportunities to revisit information throughout the semester, or even in future semesters. This does involve some up-front planning, and can be difficult to achieve, given time constraints and the need to cover a set curriculum. However, spacing can be achieved with no great costs if teachers set aside a few minutes per class to review information from previous lessons. The second method involves putting the onus to space on the students themselves. Of course, this would work best with older students – high school and above. Because spacing requires advance planning, it is crucial that the teacher helps students plan their studying. For example, teachers could suggest that students schedule study sessions on days that alternate with the days on which a particular class meets (e.g., schedule review sessions for Tuesday and Thursday when the class meets Monday and Wednesday; see Fig.  1 for a more complete weekly spaced practice schedule). It important to note that the spacing effect refers to information that is repeated multiple times, rather than the idea of studying different material in one long session versus spaced out in small study sessions over time. However, for teachers and particularly for students planning a study schedule, the subtle difference between the two situations (spacing out restudy opportunities, versus spacing out studying of different information over time) may be lost. Future research should address the effects of spacing out studying of different information over time, whether the same considerations apply in this situation as compared to spacing out restudy opportunities, and how important it is for teachers and students to understand the difference between these two types of spaced practice.

It is important to note that students may feel less confident when they space their learning (Bjork, 1999 ) than when they cram. This is because spaced learning is harder – but it is this “desirable difficulty” that helps learning in the long term (Bjork, 1994 ). Students tend to cram for exams rather than space out their learning. One explanation for this is that cramming does “work”, if the goal is only to pass an exam. In order to change students’ minds about how they schedule their studying, it might be important to emphasize the value of retaining information beyond a final exam in one course.

Ideas for how to apply spaced practice in teaching have appeared in numerous teacher blogs (e.g., Fawcett, 2013 ; Kraft, 2015 ; Picciotto, 2009 ). In England in particular, as of 2013, high-school students need to be able to remember content from up to 3 years back on cumulative exams (General Certificate of Secondary Education (GCSE) and A-level exams; see CIFE, 2012 ). A-levels in particular determine what subject students study in university and which programs they are accepted into, and thus shape the path of their academic career. A common approach for dealing with these exams has been to include a “revision” (i.e., studying or cramming) period of a few weeks leading up to the high-stakes cumulative exams. Now, teachers who follow cognitive psychology are advocating a shift of priorities to spacing learning over time across the 3 years, rather than teaching a topic once and then intensely reviewing it weeks before the exam (Cox, 2016a ; Wood, 2017 ). For example, some teachers have suggested using homework assignments as an opportunity for spaced practice by giving students homework on previous topics (Rose, 2014 ). However, questions remain, such as whether spaced practice can ever be effective enough to completely alleviate the need or utility of a cramming period (Cox, 2016b ), and how one can possibly figure out the optimal lag for spacing (Benney, 2016 ; Firth, 2016 ).

There has been considerable research on the question of optimal lag, and much of it is quite complex; two sessions neither too close together (i.e., cramming) nor too far apart are ideal for retention. In a large-scale study, Cepeda, Vul, Rohrer, Wixted, and Pashler ( 2008 ) examined the effects of the gap between study sessions and the interval between study and test across long periods, and found that the optimal gap between study sessions was contingent on the retention interval. Thus, it is not clear how teachers can apply the complex findings on lag to their own classrooms.

A useful avenue of research would be to simplify the research paradigms that are used to study optimal lag, with the goal of creating a flexible, spaced-practice framework that teachers could apply and tailor to their own teaching needs. For example, an Excel macro spreadsheet was recently produced to help teachers plan for lagged lessons (Weinstein-Jones & Weinstein, 2017 ; see Weinstein & Weinstein-Jones ( 2017 ) for a description of the algorithm used in the spreadsheet), and has been used by teachers to plan their lessons (Penfound, 2017 ). However, one teacher who found this tool helpful also wondered whether the more sophisticated plan was any better than his own method of manually selecting poorly understood material from previous classes for later review (Lovell, 2017 ). This direction is being actively explored within personalized online learning environments (Kornell & Finn, 2016 ; Lindsey, Shroyer, Pashler, & Mozer, 2014 ), but teachers in physical classrooms might need less technologically-driven solutions to teach cohorts of students.

It seems teachers would greatly appreciate a set of guidelines for how to implement spacing in the curriculum in the most effective, but also the most efficient manner. While the cognitive field has made great advances in terms of understanding the mechanisms behind spacing, what teachers need more of are concrete evidence-based tools and guidelines for direct implementation in the classroom. These could include more sophisticated and experimentally tested versions of the software described above (Weinstein-Jones & Weinstein, 2017 ), or adaptable templates of spaced curricula. Moreover, researchers need to evaluate the effectiveness of these tools in a real classroom environment, over a semester or academic year, in order to give pedagogically relevant evidence-based recommendations to teachers.

Interleaving

Another scheduling technique that has been shown to increase learning is interleaving. Interleaving occurs when different ideas or problem types are tackled in a sequence, as opposed to the more common method of attempting multiple versions of the same problem in a given study session (known as blocking). Interleaving as a principle can be applied in many different ways. One such way involves interleaving different types of problems during learning, which is particularly applicable to subjects such as math and physics (see Fig.  2 a for an example with fractions, based on a study by Patel, Liu, & Koedinger, 2016 ). For example, in a study with college students, Rohrer and Taylor ( 2007 ) found that shuffling math problems that involved calculating the volume of different shapes resulted in better test performance 1 week later than when students answered multiple problems about the same type of shape in a row. This pattern of results has also been replicated with younger students, for example 7 th grade students learning to solve graph and slope problems (Rohrer, Dedrick, & Stershic, 2015 ). The proposed explanation for the benefit of interleaving is that switching between different problem types allows students to acquire the ability to choose the right method for solving different types of problems rather than learning only the method itself, and not when to apply it.

Do the benefits of interleaving extend beyond problem solving? The answer appears to be yes. Interleaving can be helpful in other situations that require discrimination, such as inductive learning. Kornell and Bjork ( 2008 ) examined the effects of interleaving in a task that might be pertinent to a student of the history of art: the ability to match paintings to their respective painters. Students who studied different painters’ paintings interleaved at study were more successful on a later identification test than were participants who studied the paintings blocked by painter. Birnbaum, Kornell, Bjork, and Bjork ( 2013 ) proposed the discriminative-contrast hypothesis to explain that interleaving enhances learning by allowing the comparison between exemplars of different categories. They found support for this hypothesis in a set of experiments with bird categorization: participants benefited from interleaving and also from spacing, but not when the spacing interrupted side-by-side comparisons of birds from different categories.

Another type of interleaving involves the interleaving of study and test opportunities. This type of interleaving has been applied, once again, to problem solving, whereby students alternate between attempting a problem and viewing a worked example (Trafton & Reiser, 1993 ); this pattern appears to be superior to answering a string of problems in a row, at least with respect to the amount of time it takes to achieve mastery of a procedure (Corbett, Reed, Hoffmann, MacLaren, & Wagner, 2010 ). The benefits of interleaving study and test opportunities – rather than blocking study followed by attempting to answer problems or questions – might arise due to a process known as “test-potentiated learning”. That is, a study opportunity that immediately follows a retrieval attempt may be more fruitful than when that same studying was not preceded by retrieval (Arnold & McDermott, 2013 ).

For problem-based subjects, the interleaving technique is straightforward: simply mix questions on homework and quizzes with previous materials (which takes care of spacing as well); for languages, mix vocabulary themes rather than blocking by theme (Thomson & Mehring, 2016 ). But interleaving as an educational strategy ought to be presented to teachers with some caveats. Research has focused on interleaving material that is somewhat related (e.g., solving different mathematical equations, Rohrer et al., 2015 ), whereas students sometimes ask whether they should interleave material from different subjects – a practice that has not received empirical support (Hausman & Kornell, 2014 ). When advising students how to study independently, teachers should thus proceed with caution. Since it is easy for younger students to confuse this type of unhelpful interleaving with the more helpful interleaving of related information, it may be best for teachers of younger grades to create opportunities for interleaving in homework and quiz assignments rather than putting the onus on the students themselves to make use of the technique. Technology can be very helpful here, with apps such as Quizlet, Memrise, Anki, Synap, Quiz Champ, and many others (see also “Learning Scientists”, 2017 ) that not only allow instructor-created quizzes to be taken by students, but also provide built-in interleaving algorithms so that the burden does not fall on the teacher or the student to carefully plan which items are interleaved when.

An important point to consider is that in educational practice, the distinction between spacing and interleaving can be difficult to delineate. The gap between the scientific and classroom definitions of interleaving is demonstrated by teachers’ own writings about this technique. When they write about interleaving, teachers often extend the term to connote a curriculum that involves returning to topics multiple times throughout the year (e.g., Kirby, 2014 ; see “Learning Scientists” ( 2016a ) for a collection of similar blog posts by several other teachers). The “interleaving” of topics throughout the curriculum produces an effect that is more akin to what cognitive psychologists call “spacing” (see Fig.  2 b for a visual representation of the difference between interleaving and spacing). However, cognitive psychologists have not examined the effects of structuring the curriculum in this way, and open questions remain: does repeatedly circling back to previous topics throughout the semester interrupt the learning of new information? What are some effective techniques for interleaving old and new information within one class? And how does one determine the balance between old and new information?

Retrieval practice

While tests are most often used in educational settings for assessment, a lesser-known benefit of tests is that they actually improve memory of the tested information. If we think of our memories as libraries of information, then it may seem surprising that retrieval (which happens when we take a test) improves memory; however, we know from a century of research that retrieving knowledge actually strengthens it (see Karpicke, Lehman, & Aue, 2014 ). Testing was shown to strengthen memory as early as 100 years ago (Gates, 1917 ), and there has been a surge of research in the last decade on the mnemonic benefits of testing, or retrieval practice . Most of the research on the effectiveness of retrieval practice has been done with college students (see Roediger & Karpicke, 2006 ; Roediger, Putnam, & Smith, 2011 ), but retrieval-based learning has been shown to be effective at producing learning for a wide range of ages, including preschoolers (Fritz, Morris, Nolan, & Singleton, 2007 ), elementary-aged children (e.g., Karpicke, Blunt, & Smith, 2016 ; Karpicke, Blunt, Smith, & Karpicke, 2014 ; Lipko-Speed, Dunlosky, & Rawson, 2014 ; Marsh, Fazio, & Goswick, 2012 ; Ritchie, Della Sala, & McIntosh, 2013 ), middle-school students (e.g., McDaniel, Thomas, Agarwal, McDermott, & Roediger, 2013 ; McDermott, Agarwal, D’Antonio, Roediger, & McDaniel, 2014 ), and high-school students (e.g., McDermott et al., 2014 ). In addition, the effectiveness of retrieval-based learning has been extended beyond simple testing to other activities in which retrieval practice can be integrated, such as concept mapping (Blunt & Karpicke, 2014 ; Karpicke, Blunt, et al., 2014 ; Ritchie et al., 2013 ).

A debate is currently ongoing as to the effectiveness of retrieval practice for more complex materials (Karpicke & Aue, 2015 ; Roelle & Berthold, 2017 ; Van Gog & Sweller, 2015 ). Practicing retrieval has been shown to improve the application of knowledge to new situations (e.g., Butler, 2010 ; Dirkx, Kester, & Kirschner, 2014 ); McDaniel et al., 2013 ; Smith, Blunt, Whiffen, & Karpicke, 2016 ); but see Tran, Rohrer, and Pashler ( 2015 ) and Wooldridge, Bugg, McDaniel, and Liu ( 2014 ), for retrieval practice studies that showed limited or no increased transfer compared to restudy. Retrieval practice effects on higher-order learning may be more sensitive than fact learning to encoding factors, such as the way material is presented during study (Eglington & Kang, 2016 ). In addition, retrieval practice may be more beneficial for higher-order learning if it includes more scaffolding (Fiechter & Benjamin, 2017 ; but see Smith, Blunt, et al., 2016 ) and targeted practice with application questions (Son & Rivas, 2016 ).

How does retrieval practice help memory? Figure  3 illustrates both the direct and indirect benefits of retrieval practice identified by the literature. The act of retrieval itself is thought to strengthen memory (Karpicke, Blunt, et al., 2014 ; Roediger & Karpicke, 2006 ; Smith, Roediger, & Karpicke, 2013 ). For example, Smith et al. ( 2013 ) showed that if students brought information to mind without actually producing it (covert retrieval), they remembered the information just as well as if they overtly produced the retrieved information (overt retrieval). Importantly, both overt and covert retrieval practice improved memory over control groups without retrieval practice, even when feedback was not provided. The fact that bringing information to mind in the absence of feedback or restudy opportunities improves memory leads researchers to conclude that it is the act of retrieval – thinking back to bring information to mind – that improves memory of that information.

The benefit of retrieval practice depends to a certain extent on successful retrieval (see Karpicke, Lehman, et al., 2014 ). For example, in Experiment 4 of Smith et al. ( 2013 ), students successfully retrieved 72% of the information during retrieval practice. Of course, retrieving 72% of the information was compared to a restudy control group, during which students were re-exposed to 100% of the information, creating a bias in favor of the restudy condition. Yet retrieval led to superior memory later compared to the restudy control. However, if retrieval success is extremely low, then it is unlikely to improve memory (e.g., Karpicke, Blunt, et al., 2014 ), particularly in the absence of feedback. On the other hand, if retrieval-based learning situations are constructed in such a way that ensures high levels of success, the act of bringing the information to mind may be undermined, thus making it less beneficial. For example, if a student reads a sentence and then immediately covers the sentence and recites it out loud, they are likely not retrieving the information but rather just keeping the information in their working memory long enough to recite it again (see Smith, Blunt, et al., 2016 for a discussion of this point). Thus, it is important to balance success of retrieval with overall difficulty in retrieving the information (Smith & Karpicke, 2014 ; Weinstein, Nunes, & Karpicke, 2016 ). If initial retrieval success is low, then feedback can help improve the overall benefit of practicing retrieval (Kang, McDermott, & Roediger, 2007 ; Smith & Karpicke, 2014 ). Kornell, Klein, and Rawson ( 2015 ), however, found that it was the retrieval attempt and not the correct production of information that produced the retrieval practice benefit – as long as the correct answer was provided after an unsuccessful attempt, the benefit was the same as for a successful retrieval attempt in this set of studies. From a practical perspective, it would be helpful for teachers to know when retrieval attempts in the absence of success are helpful, and when they are not. There may also be additional reasons beyond retrieval benefits that would push teachers towards retrieval practice activities that produce some success amongst students; for example, teachers may hesitate to give students retrieval practice exercises that are too difficult, as this may negatively affect self-efficacy and confidence.

In addition to the fact that bringing information to mind directly improves memory for that information, engaging in retrieval practice can produce indirect benefits as well (see Roediger et al., 2011 ). For example, research by Weinstein, Gilmore, Szpunar, and McDermott ( 2014 ) demonstrated that when students expected to be tested, the increased test expectancy led to better-quality encoding of new information. Frequent testing can also serve to decrease mind-wandering – that is, thoughts that are unrelated to the material that students are supposed to be studying (Szpunar, Khan, & Schacter, 2013 ).

Practicing retrieval is a powerful way to improve meaningful learning of information, and it is relatively easy to implement in the classroom. For example, requiring students to practice retrieval can be as simple as asking students to put their class materials away and try to write out everything they know about a topic. Retrieval-based learning strategies are also flexible. Instructors can give students practice tests (e.g., short-answer or multiple-choice, see Smith & Karpicke, 2014 ), provide open-ended prompts for the students to recall information (e.g., Smith, Blunt, et al., 2016 ) or ask their students to create concept maps from memory (e.g., Blunt & Karpicke, 2014 ). In one study, Weinstein et al. ( 2016 ) looked at the effectiveness of inserting simple short-answer questions into online learning modules to see whether they improved student performance. Weinstein and colleagues also manipulated the placement of the questions. For some students, the questions were interspersed throughout the module, and for other students the questions were all presented at the end of the module. Initial success on the short-answer questions was higher when the questions were interspersed throughout the module. However, on a later test of learning from that module, the original placement of the questions in the module did not matter for performance. As with spaced practice, where the optimal gap between study sessions is contingent on the retention interval, the optimum difficulty and level of success during retrieval practice may also depend on the retention interval. Both groups of students who answered questions performed better on the delayed test compared to a control group without question opportunities during the module. Thus, the important thing is for instructors to provide opportunities for retrieval practice during learning. Based on previous research, any activity that promotes the successful retrieval of information should improve learning.

Retrieval practice has received a lot of attention in teacher blogs (see “Learning Scientists” ( 2016b ) for a collection). A common theme seems to be an emphasis on low-stakes (Young, 2016 ) and even no-stakes (Cox, 2015 ) testing, the goal of which is to increase learning rather than assess performance. In fact, one well-known charter school in the UK has an official homework policy grounded in retrieval practice: students are to test themselves on subject knowledge for 30 minutes every day in lieu of standard homework (Michaela Community School, 2014 ). The utility of homework, particularly for younger children, is often a hotly debated topic outside of academia (e.g., Shumaker, 2016 ; but see Jones ( 2016 ) for an opposing viewpoint and Cooper ( 1989 ) for the original research the blog posts were based on). Whereas some research shows clear links between homework and academic achievement (Valle et al., 2016 ), other researchers have questioned the effectiveness of homework (Dettmers, Trautwein, & Lüdtke, 2009 ). Perhaps amending homework to involve retrieval practice might make it more effective; this remains an open empirical question.

One final consideration is that of test anxiety. While retrieval practice can be very powerful at improving memory, some research shows that pressure during retrieval can undermine some of the learning benefit. For example, Hinze and Rapp ( 2014 ) manipulated pressure during quizzing to create high-pressure and low-pressure conditions. On the quizzes themselves, students performed equally well. However, those in the high-pressure condition did not perform as well on a criterion test later compared to the low-pressure group. Thus, test anxiety may reduce the learning benefit of retrieval practice. Eliminating all high-pressure tests is probably not possible, but instructors can provide a number of low-stakes retrieval opportunities for students to help increase learning. The use of low-stakes testing can serve to decrease test anxiety (Khanna, 2015 ), and has recently been shown to negate the detrimental impact of stress on learning (Smith, Floerke, & Thomas, 2016 ). This is a particularly important line of inquiry to pursue for future research, because many teachers who are not familiar with the effectiveness of retrieval practice may be put off by the implied pressure of “testing”, which evokes the much maligned high-stakes standardized tests (e.g., McHugh, 2013 ).

Elaboration

Elaboration involves connecting new information to pre-existing knowledge. Anderson ( 1983 , p.285) made the following claim about elaboration: “One of the most potent manipulations that can be performed in terms of increasing a subject’s memory for material is to have the subject elaborate on the to-be-remembered material.” Postman ( 1976 , p. 28) defined elaboration most parsimoniously as “additions to nominal input”, and Hirshman ( 2001 , p. 4369) provided an elaboration on this definition (pun intended!), defining elaboration as “A conscious, intentional process that associates to-be-remembered information with other information in memory.” However, in practice, elaboration could mean many different things. The common thread in all the definitions is that elaboration involves adding features to an existing memory.

One possible instantiation of elaboration is thinking about information on a deeper level. The levels (or “depth”) of processing framework, proposed by Craik and Lockhart ( 1972 ), predicts that information will be remembered better if it is processed more deeply in terms of meaning, rather than shallowly in terms of form. The leves of processing framework has, however, received a number of criticisms (Craik, 2002 ). One major problem with this framework is that it is difficult to measure “depth”. And if we are not able to actually measure depth, then the argument can become circular: is it that something was remembered better because it was studied more deeply, or do we conclude that it must have been studied more deeply because it is remembered better? (See Lockhart & Craik, 1990 , for further discussion of this issue).

Another mechanism by which elaboration can confer a benefit to learning is via improvement in organization (Bellezza, Cheesman, & Reddy, 1977 ; Mandler, 1979 ). By this view, elaboration involves making information more integrated and organized with existing knowledge structures. By connecting and integrating the to-be-learned information with other concepts in memory, students can increase the extent to which the ideas are organized in their minds, and this increased organization presumably facilitates the reconstruction of the past at the time of retrieval.

Elaboration is such a broad term and can include so many different techniques that it is hard to claim that elaboration will always help learning. There is, however, a specific technique under the umbrella of elaboration for which there is relatively strong evidence in terms of effectiveness (Dunlosky et al., 2013 ; Pashler et al., 2007 ). This technique is called elaborative interrogation, and involves students questioning the materials that they are studying (Pressley, McDaniel, Turnure, Wood, & Ahmad, 1987 ). More specifically, students using this technique would ask “how” and “why” questions about the concepts they are studying (see Fig.  4 for an example on the physics of flight). Then, crucially, students would try to answer these questions – either from their materials or, eventually, from memory (McDaniel & Donnelly, 1996 ). The process of figuring out the answer to the questions – with some amount of uncertainty (Overoye & Storm, 2015 ) – can help learning. When using this technique, however, it is important that students check their answers with their materials or with the teacher; when the content generated through elaborative interrogation is poor, it can actually hurt learning (Clinton, Alibali, & Nathan, 2016 ).

Students can also be encouraged to self-explain concepts to themselves while learning (Chi, De Leeuw, Chiu, & LaVancher, 1994 ). This might involve students simply saying out loud what steps they need to perform to solve an equation. Aleven and Koedinger ( 2002 ) conducted two classroom studies in which students were either prompted by a “cognitive tutor” to provide self-explanations during a problem-solving task or not, and found that the self-explanations led to improved performance. According to the authors, this approach could scale well to real classrooms. If possible and relevant, students could even perform actions alongside their self-explanations (Cohen, 1981 ; see also the enactment effect, Hainselin, Picard, Manolli, Vankerkore-Candas, & Bourdin, 2017 ). Instructors can scaffold students in these types of activities by providing self-explanation prompts throughout to-be-learned material (O’Neil et al., 2014 ). Ultimately, the greatest potential benefit of accurate self-explanation or elaboration is that the student will be able to transfer their knowledge to a new situation (Rittle-Johnson, 2006 ).

The technical term “elaborative interrogation” has not made it into the vernacular of educational bloggers (a search on https://educationechochamberuncut.wordpress.com , which consolidates over 3,000 UK-based teacher blogs, yielded zero results for that term). However, a few teachers have blogged about elaboration more generally (e.g., Hobbiss, 2016 ) and deep questioning specifically (e.g., Class Teaching, 2013 ), just without using the specific terminology. This strategy in particular may benefit from a more open dialog between researchers and teachers to facilitate the use of elaborative interrogation in the classroom and to address possible barriers to implementation. In terms of advancing the scientific understanding of elaborative interrogation in a classroom setting, it would be informative to conduct a larger-scale intervention to see whether having students elaborate during reading actually helps their understanding. It would also be useful to know whether the students really need to generate their own elaborative interrogation (“how” and “why”) questions, versus answering questions provided by others. How long should students persist to find the answers? When is the right time to have students engage in this task, given the levels of expertise required to do it well (Clinton et al., 2016 )? Without knowing the answers to these questions, it may be too early for us to instruct teachers to use this technique in their classes. Finally, elaborative interrogation takes a long time. Is this time efficiently spent? Or, would it be better to have the students try to answer a few questions, pool their information as a class, and then move to practicing retrieval of the information?

Concrete examples

Providing supporting information can improve the learning of key ideas and concepts. Specifically, using concrete examples to supplement content that is more conceptual in nature can make the ideas easier to understand and remember. Concrete examples can provide several advantages to the learning process: (a) they can concisely convey information, (b) they can provide students with more concrete information that is easier to remember, and (c) they can take advantage of the superior memorability of pictures relative to words (see “Dual Coding”).

Words that are more concrete are both recognized and recalled better than abstract words (Gorman, 1961 ; e.g., “button” and “bound,” respectively). Furthermore, it has been demonstrated that information that is more concrete and imageable enhances the learning of associations, even with abstract content (Caplan & Madan, 2016 ; Madan, Glaholt, & Caplan, 2010 ; Paivio, 1971 ). Following from this, providing concrete examples during instruction should improve retention of related abstract concepts, rather than the concrete examples alone being remembered better. Concrete examples can be useful both during instruction and during practice problems. Having students actively explain how two examples are similar and encouraging them to extract the underlying structure on their own can also help with transfer. In a laboratory study, Berry ( 1983 ) demonstrated that students performed well when given concrete practice problems, regardless of the use of verbalization (akin to elaborative interrogation), but that verbalization helped students transfer understanding from concrete to abstract problems. One particularly important area of future research is determining how students can best make the link between concrete examples and abstract ideas.

Since abstract concepts are harder to grasp than concrete information (Paivio, Walsh, & Bons, 1994 ), it follows that teachers ought to illustrate abstract ideas with concrete examples. However, care must be taken when selecting the examples. LeFevre and Dixon ( 1986 ) provided students with both concrete examples and abstract instructions and found that when these were inconsistent, students followed the concrete examples rather than the abstract instructions, potentially constraining the application of the abstract concept being taught. Lew, Fukawa-Connelly, Mejí-Ramos, and Weber ( 2016 ) used an interview approach to examine why students may have difficulty understanding a lecture. Responses indicated that some issues were related to understanding the overarching topic rather than the component parts, and to the use of informal colloquialisms that did not clearly follow from the material being taught. Both of these issues could have potentially been addressed through the inclusion of a greater number of relevant concrete examples.

One concern with using concrete examples is that students might only remember the examples – especially if they are particularly memorable, such as fun or gimmicky examples – and will not be able to transfer their understanding from one example to another, or more broadly to the abstract concept. However, there does not seem to be any evidence that fun relevant examples actually hurt learning by harming memory for important information. Instead, fun examples and jokes tend to be more memorable, but this boost in memory for the joke does not seem to come at a cost to memory for the underlying concept (Baldassari & Kelley, 2012 ). However, two important caveats need to be highlighted. First, to the extent that the more memorable content is not relevant to the concepts of interest, learning of the target information can be compromised (Harp & Mayer, 1998 ). Thus, care must be taken to ensure that all examples and gimmicks are, in fact, related to the core concepts that the students need to acquire, and do not contain irrelevant perceptual features (Kaminski & Sloutsky, 2013 ).

The second issue is that novices often notice and remember the surface details of an example rather than the underlying structure. Experts, on the other hand, can extract the underlying structure from examples that have divergent surface features (Chi, Feltovich, & Glaser, 1981 ; see Fig.  5 for an example from physics). Gick and Holyoak ( 1983 ) tried to get students to apply a rule from one problem to another problem that appeared different on the surface, but was structurally similar. They found that providing multiple examples helped with this transfer process compared to only using one example – especially when the examples provided had different surface details. More work is also needed to determine how many examples are sufficient for generalization to occur (and this, of course, will vary with contextual factors and individual differences). Further research on the continuum between concrete/specific examples and more abstract concepts would also be informative. That is, if an example is not concrete enough, it may be too difficult to understand. On the other hand, if the example is too concrete, that could be detrimental to generalization to the more abstract concept (although a diverse set of very concrete examples may be able to help with this). In fact, in a controversial article, Kaminski, Sloutsky, and Heckler ( 2008 ) claimed that abstract examples were more effective than concrete examples. Later rebuttals of this paper contested whether the abstract versus concrete distinction was clearly defined in the original study (see Reed, 2008 , for a collection of letters on the subject). This ideal point along the concrete-abstract continuum might also interact with development.

Finding teacher blog posts on concrete examples proved to be more difficult than for the other strategies in this review. One optimistic possibility is that teachers frequently use concrete examples in their teaching, and thus do not think of this as a specific contribution from cognitive psychology; the one blog post we were able to find that discussed concrete examples suggests that this might be the case (Boulton, 2016 ). The idea of “linking abstract concepts with concrete examples” is also covered in 25% of teacher-training textbooks used in the US, according to the report by Pomerance et al. ( 2016 ); this is the second most frequently covered of the six strategies, after “posing probing questions” (i.e., elaborative interrogation). A useful direction for future research would be to establish how teachers are using concrete examples in their practice, and whether we can make any suggestions for improvement based on research into the science of learning. For example, if two examples are better than one (Bauernschmidt, 2017 ), are additional examples also needed, or are there diminishing returns from providing more examples? And, how can teachers best ensure that concrete examples are consistent with prior knowledge (Reed, 2008 )?

Dual coding

Both the memory literature and folk psychology support the notion of visual examples being beneficial—the adage of “a picture is worth a thousand words” (traced back to an advertising slogan from the 1920s; Meider, 1990 ). Indeed, it is well-understood that more information can be conveyed through a simple illustration than through several paragraphs of text (e.g., Barker & Manji, 1989 ; Mayer & Gallini, 1990 ). Illustrations can be particularly helpful when the described concept involves several parts or steps and is intended for individuals with low prior knowledge (Eitel & Scheiter, 2015 ; Mayer & Gallini, 1990 ). Figure  6 provides a concrete example of this, illustrating how information can flow through neurons and synapses.

In addition to being able to convey information more succinctly, pictures are also more memorable than words (Paivio & Csapo, 1969 , 1973 ). In the memory literature, this is referred to as the picture superiority effect , and dual coding theory was developed in part to explain this effect. Dual coding follows from the notion of text being accompanied by complementary visual information to enhance learning. Paivio ( 1971 , 1986 ) proposed dual coding theory as a mechanistic account for the integration of multiple information “codes” to process information. In this theory, a code corresponds to a modal or otherwise distinct representation of a concept—e.g., “mental images for ‘book’ have visual, tactual, and other perceptual qualities similar to those evoked by the referent objects on which the images are based” (Clark & Paivio, 1991 , p. 152). Aylwin ( 1990 ) provides a clear example of how the word “dog” can evoke verbal, visual, and enactive representations (see Fig.  7 for a similar example for the word “SPOON”, based on Aylwin, 1990 (Fig.  2 ) and Madan & Singhal, 2012a (Fig.  3 )). Codes can also correspond to emotional properties (Clark & Paivio, 1991 ; Paivio, 2013 ). Clark and Paivio ( 1991 ) provide a thorough review of dual coding theory and its relation to education, while Paivio ( 2007 ) provides a comprehensive treatise on dual coding theory. Broadly, dual coding theory suggests that providing multiple representations of the same information enhances learning and memory, and that information that more readily evokes additional representations (through automatic imagery processes) receives a similar benefit.

Paivio and Csapo ( 1973 ) suggest that verbal and imaginal codes have independent and additive effects on memory recall. Using visuals to improve learning and memory has been particularly applied to vocabulary learning (Danan, 1992 ; Sadoski, 2005 ), but has also shown success in other domains such as in health care (Hartland, Biddle, & Fallacaro, 2008 ). To take advantage of dual coding, verbal information should be accompanied by a visual representation when possible. However, while the studies discussed all indicate that the use of multiple representations of information is favorable, it is important to acknowledge that each representation also increases cognitive load and can lead to over-saturation (Mayer & Moreno, 2003 ).

Given that pictures are generally remembered better than words, it is important to ensure that the pictures students are provided with are helpful and relevant to the content they are expected to learn. McNeill, Uttal, Jarvin, and Sternberg ( 2009 ) found that providing visual examples decreased conceptual errors. However, McNeill et al. also found that when students were given visually rich examples, they performed more poorly than students who were not given any visual example, suggesting that the visual details can at times become a distraction and hinder performance. Thus, it is important to consider that images used in teaching are clear and not ambiguous in their meaning (Schwartz, 2007 ).

Further broadening the scope of dual coding theory, Engelkamp and Zimmer ( 1984 ) suggest that motor movements, such as “turning the handle,” can provide an additional motor code that can improve memory, linking studies of motor actions (enactment) with dual coding theory (Clark & Paivio, 1991 ; Engelkamp & Cohen, 1991 ; Madan & Singhal, 2012c ). Indeed, enactment effects appear to primarily occur during learning, rather than during retrieval (Peterson & Mulligan, 2010 ). Along similar lines, Wammes, Meade, and Fernandes ( 2016 ) demonstrated that generating drawings can provide memory benefits beyond what could otherwise be explained by visual imagery, picture superiority, and other memory enhancing effects. Providing convergent evidence, even when overt motor actions are not critical in themselves, words representing functional objects have been shown to enhance later memory (Madan & Singhal, 2012b ; Montefinese, Ambrosini, Fairfield, & Mammarella, 2013 ). This indicates that motoric processes can improve memory similarly to visual imagery, similar to memory differences for concrete vs. abstract words. Further research suggests that automatic motor simulation for functional objects is likely responsible for this memory benefit (Madan, Chen, & Singhal, 2016 ).

When teachers combine visuals and words in their educational practice, however, they may not always be taking advantage of dual coding – at least, not in the optimal manner. For example, a recent discussion on Twitter centered around one teacher’s decision to have 7 th Grade students replace certain words in their science laboratory report with a picture of that word (e.g., the instructions read “using a syringe …” and a picture of a syringe replaced the word; Turner, 2016a ). Other teachers argued that this was not dual coding (Beaven, 2016 ; Williams, 2016 ), because there were no longer two different representations of the information. The first teacher maintained that dual coding was preserved, because this laboratory report with pictures was to be used alongside the original, fully verbal report (Turner, 2016b ). This particular implementation – having students replace individual words with pictures – has not been examined in the cognitive literature, presumably because no benefit would be expected. In any case, we need to be clearer about implementations for dual coding, and more research is needed to clarify how teachers can make use of the benefits conferred by multiple representations and picture superiority.

Critically, dual coding theory is distinct from the notion of “learning styles,” which describe the idea that individuals benefit from instruction that matches their modality preference. While this idea is pervasive and individuals often subjectively feel that they have a preference, evidence indicates that the learning styles theory is not supported by empirical findings (e.g., Kavale, Hirshoren, & Forness, 1998 ; Pashler, McDaniel, Rohrer, & Bjork, 2008 ; Rohrer & Pashler, 2012 ). That is, there is no evidence that instructing students in their preferred learning style leads to an overall improvement in learning (the “meshing” hypothesis). Moreover, learning styles have come to be described as a myth or urban legend within psychology (Coffield, Moseley, Hall, & Ecclestone, 2004 ; Hattie & Yates, 2014 ; Kirschner & van Merriënboer, 2013 ; Kirschner, 2017 ); skepticism about learning styles is a common stance amongst evidence-informed teachers (e.g., Saunders, 2016 ). Providing evidence against the notion of learning styles, Kraemer, Rosenberg, and Thompson-Schill ( 2009 ) found that individuals who scored as “verbalizers” and “visualizers” did not perform any better on experimental trials matching their preference. Instead, it has recently been shown that learning through one’s preferred learning style is associated with elevated subjective judgements of learning, but not objective performance (Knoll, Otani, Skeel, & Van Horn, 2017 ). In contrast to learning styles, dual coding is based on providing additional, complementary forms of information to enhance learning, rather than tailoring instruction to individuals’ preferences.

Genuine educational environments present many opportunities for combining the strategies outlined above. Spacing can be particularly potent for learning if it is combined with retrieval practice. The additive benefits of retrieval practice and spacing can be gained by engaging in retrieval practice multiple times (also known as distributed practice; see Cepeda et al., 2006 ). Interleaving naturally entails spacing if students interleave old and new material. Concrete examples can be both verbal and visual, making use of dual coding. In addition, the strategies of elaboration, concrete examples, and dual coding all work best when used as part of retrieval practice. For example, in the concept-mapping studies mentioned above (Blunt & Karpicke, 2014 ; Karpicke, Blunt, et al., 2014 ), creating concept maps while looking at course materials (e.g., a textbook) was not as effective for later memory as creating concept maps from memory. When practicing elaborative interrogation, students can start off answering the “how” and “why” questions they pose for themselves using class materials, and work their way up to answering them from memory. And when interleaving different problem types, students should be practicing answering them rather than just looking over worked examples.

But while these ideas for strategy combinations have empirical bases, it has not yet been established whether the benefits of the strategies to learning are additive, super-additive, or, in some cases, incompatible. Thus, future research needs to (a) better formalize the definition of each strategy (particularly critical for elaboration and dual coding), (b) identify best practices for implementation in the classroom, (c) delineate the boundary conditions of each strategy, and (d) strategically investigate interactions between the six strategies we outlined in this manuscript.

Aleven, V. A., & Koedinger, K. R. (2002). An effective metacognitive strategy: learning by doing and explaining with a computer-based cognitive tutor. Cognitive Science, 26 , 147–179.

Article   Google Scholar  

Anderson, J. R. (1983). A spreading activation theory of memory. Journal of Verbal Learning and Verbal Behavior, 22 , 261–295.

Arnold, K. M., & McDermott, K. B. (2013). Test-potentiated learning: distinguishing between direct and indirect effects of tests. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39 , 940–945.

PubMed   Google Scholar  

Aylwin, S. (1990). Imagery and affect: big questions, little answers. In P. J. Thompson, D. E. Marks, & J. T. E. Richardson (Eds.), Imagery: Current developments . New York: International Library of Psychology.

Google Scholar  

Baldassari, M. J., & Kelley, M. (2012). Make’em laugh? The mnemonic effect of humor in a speech. Psi Chi Journal of Psychological Research, 17 , 2–9.

Barker, P. G., & Manji, K. A. (1989). Pictorial dialogue methods. International Journal of Man-Machine Studies, 31 , 323–347.

Bauernschmidt, A. (2017). GUEST POST: two examples are better than one. [Blog post]. The Learning Scientists Blog . Retrieved from http://www.learningscientists.org/blog/2017/5/30-1 . Accessed 25 Dec 2017.

Beaven, T. (2016). @doctorwhy @FurtherEdagogy @doc_kristy Right, I thought the whole point of dual coding was to use TWO codes: pics + words of the SAME info? [Tweet]. Retrieved from https://twitter.com/TitaBeaven/status/807504041341308929 . Accessed 25 Dec 2017.

Bellezza, F. S., Cheesman, F. L., & Reddy, B. G. (1977). Organization and semantic elaboration in free recall. Journal of Experimental Psychology: Human Learning and Memory, 3 , 539–550.

Benney, D. (2016). (Trying to apply) spacing in a content heavy subject [Blog post]. Retrieved from https://mrbenney.wordpress.com/2016/10/16/trying-to-apply-spacing-in-science/ . Accessed 25 Dec 2017.

Berry, D. C. (1983). Metacognitive experience and transfer of logical reasoning. Quarterly Journal of Experimental Psychology, 35A , 39–49.

Birnbaum, M. S., Kornell, N., Bjork, E. L., & Bjork, R. A. (2013). Why interleaving enhances inductive learning: the roles of discrimination and retrieval. Memory & Cognition, 41 , 392–402.

Bjork, R. A. (1999). Assessing our own competence: heuristics and illusions. In D. Gopher & A. Koriat (Eds.), Attention and peformance XVII. Cognitive regulation of performance: Interaction of theory and application (pp. 435–459). Cambridge, MA: MIT Press.

Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185–205). Cambridge, MA: MIT Press.

Bjork, R. A., & Bjork, E. L. (1992). A new theory of disuse and an old theory of stimulus fluctuation. From learning processes to cognitive processes: Essays in honor of William K. Estes, 2 , 35–67.

Bjork, E. L., & Bjork, R. A. (2011). Making things hard on yourself, but in a good way: creating desirable difficulties to enhance learning. Psychology and the real world: Essays illustrating fundamental contributions to society , 56–64.

Blunt, J. R., & Karpicke, J. D. (2014). Learning with retrieval-based concept mapping. Journal of Educational Psychology, 106 , 849–858.

Boulton, K. (2016). What does cognitive overload look like in the humanities? [Blog post]. Retrieved from https://educationechochamberuncut.wordpress.com/2016/03/05/what-does-cognitive-overload-look-like-in-the-humanities-kris-boulton-2/ . Accessed 25 Dec 2017.

Brown, P. C., Roediger, H. L., & McDaniel, M. A. (2014). Make it stick . Cambridge, MA: Harvard University Press.

Book   Google Scholar  

Butler, A. C. (2010). Repeated testing produces superior transfer of learning relative to repeated studying. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36 , 1118–1133.

Caplan, J. B., & Madan, C. R. (2016). Word-imageability enhances association-memory by recruiting hippocampal activity. Journal of Cognitive Neuroscience, 28 , 1522–1538.

Article   PubMed   Google Scholar  

Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., & Rohrer, D. (2006). Distributed practice in verbal recall tasks: a review and quantitative synthesis. Psychological Bulletin, 132 , 354–380.

Cepeda, N. J., Vul, E., Rohrer, D., Wixted, J. T., & Pashler, H. (2008). Spacing effects in learning a temporal ridgeline of optimal retention. Psychological Science, 19 , 1095–1102.

Chi, M. T., De Leeuw, N., Chiu, M. H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18 , 439–477.

Chi, M. T., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5 , 121–152.

CIFE. (2012). No January A level and other changes. Retrieved from http://www.cife.org.uk/cife-general-news/no-january-a-level-and-other-changes/ . Accessed 25 Dec 2017.

Clark, D. (2016). One book on learning that every teacher, lecturer & trainer should read (7 reasons) [Blog post]. Retrieved from http://donaldclarkplanb.blogspot.com/2016/03/one-book-on-learning-that-every-teacher.html . Accessed 25 Dec 2017.

Clark, J. M., & Paivio, A. (1991). Dual coding theory and education. Educational Psychology Review, 3 , 149–210.

Class Teaching. (2013). Deep questioning [Blog post]. Retrieved from https://classteaching.wordpress.com/2013/07/12/deep-questioning/ . Accessed 25 Dec 2017.

Clinton, V., Alibali, M. W., & Nathan, M. J. (2016). Learning about posterior probability: do diagrams and elaborative interrogation help? The Journal of Experimental Education, 84 , 579–599.

Coffield, F., Moseley, D., Hall, E., & Ecclestone, K. (2004). Learning styles and pedagogy in post-16 learning: a systematic and critical review . London: Learning & Skills Research Centre.

Cohen, R. L. (1981). On the generality of some memory laws. Scandinavian Journal of Psychology, 22 , 267–281.

Cooper, H. (1989). Synthesis of research on homework. Educational Leadership, 47 , 85–91.

Corbett, A. T., Reed, S. K., Hoffmann, R., MacLaren, B., & Wagner, A. (2010). Interleaving worked examples and cognitive tutor support for algebraic modeling of problem situations. In Proceedings of the Thirty-Second Annual Meeting of the Cognitive Science Society (pp. 2882–2887).

Cox, D. (2015). No stakes testing – not telling students their results [Blog post]. Retrieved from https://missdcoxblog.wordpress.com/2015/06/06/no-stakes-testing-not-telling-students-their-results/ . Accessed 25 Dec 2017.

Cox, D. (2016a). Ditch revision. Teach it well [Blog post]. Retrieved from https://missdcoxblog.wordpress.com/2016/01/09/ditch-revision-teach-it-well/ . Accessed 25 Dec 2017.

Cox, D. (2016b). ‘They need to remember this in three years time’: spacing & interleaving for the new GCSEs [Blog post]. Retrieved from https://missdcoxblog.wordpress.com/2016/03/25/they-need-to-remember-this-in-three-years-time-spacing-interleaving-for-the-new-gcses/ . Accessed 25 Dec 2017.

Craik, F. I. (2002). Levels of processing: past, present… future? Memory, 10 , 305–318.

Craik, F. I., & Lockhart, R. S. (1972). Levels of processing: a framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11 , 671–684.

Danan, M. (1992). Reversed subtitling and dual coding theory: new directions for foreign language instruction. Language Learning, 42 , 497–527.

Dettmers, S., Trautwein, U., & Lüdtke, O. (2009). The relationship between homework time and achievement is not universal: evidence from multilevel analyses in 40 countries. School Effectiveness and School Improvement, 20 , 375–405.

Dirkx, K. J., Kester, L., & Kirschner, P. A. (2014). The testing effect for learning principles and procedures from texts. The Journal of Educational Research, 107 , 357–364.

Dunlosky, J. (2013). Strengthening the student toolbox: study strategies to boost learning. American Educator, 37 (3), 12–21.

Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’ learning with effective learning techniques: promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14 , 4–58.

Ebbinghaus, H. (1913). Memory (HA Ruger & CE Bussenius, Trans.). New York: Columbia University, Teachers College. (Original work published 1885) . Retrieved from http://psychclassics.yorku.ca/Ebbinghaus/memory8.htm . Accessed 25 Dec 2017.

Eglington, L. G., & Kang, S. H. (2016). Retrieval practice benefits deductive inference. Educational Psychology Review , 1–14.

Eitel, A., & Scheiter, K. (2015). Picture or text first? Explaining sequential effects when learning with pictures and text. Educational Psychology Review, 27 , 153–180.

Engelkamp, J., & Cohen, R. L. (1991). Current issues in memory of action events. Psychological Research, 53 , 175–182.

Engelkamp, J., & Zimmer, H. D. (1984). Motor programme information as a separable memory unit. Psychological Research, 46 , 283–299.

Fawcett, D. (2013). Can I be that little better at……using cognitive science/psychology/neurology to plan learning? [Blog post]. Retrieved from http://reflectionsofmyteaching.blogspot.com/2013/09/can-i-be-that-little-better-atusing.html . Accessed 25 Dec 2017.

Fiechter, J. L., & Benjamin, A. S. (2017). Diminishing-cues retrieval practice: a memory-enhancing technique that works when regular testing doesn’t. Psychonomic Bulletin & Review , 1–9.

Firth, J. (2016). Spacing in teaching practice [Blog post]. Retrieved from http://www.learningscientists.org/blog/2016/4/12-1 . Accessed 25 Dec 2017.

Fordham, M. [mfordhamhistory]. (2016). Is there a meaningful distinction in psychology between ‘thinking’ & ‘critical thinking’? [Tweet]. Retrieved from https://twitter.com/mfordhamhistory/status/809525713623781377 . Accessed 25 Dec 2017.

Fritz, C. O., Morris, P. E., Nolan, D., & Singleton, J. (2007). Expanding retrieval practice: an effective aid to preschool children’s learning. The Quarterly Journal of Experimental Psychology, 60 , 991–1004.

Gates, A. I. (1917). Recitation as a factory in memorizing. Archives of Psychology, 6.

Gick, M. L., & Holyoak, K. J. (1983). Schema induction and analogical transfer. Cognitive Psychology, 15 , 1–38.

Gorman, A. M. (1961). Recognition memory for nouns as a function of abstractedness and frequency. Journal of Experimental Psychology, 61 , 23–39.

Hainselin, M., Picard, L., Manolli, P., Vankerkore-Candas, S., & Bourdin, B. (2017). Hey teacher, don’t leave them kids alone: action is better for memory than reading. Frontiers in Psychology , 8 .

Harp, S. F., & Mayer, R. E. (1998). How seductive details do their damage. Journal of Educational Psychology, 90 , 414–434.

Hartland, W., Biddle, C., & Fallacaro, M. (2008). Audiovisual facilitation of clinical knowledge: A paradigm for dispersed student education based on Paivio’s dual coding theory. AANA Journal, 76 , 194–198.

Hattie, J., & Yates, G. (2014). Visible learning and the science of how we learn . New York: Routledge.

Hausman, H., & Kornell, N. (2014). Mixing topics while studying does not enhance learning. Journal of Applied Research in Memory and Cognition, 3 , 153–160.

Hinze, S. R., & Rapp, D. N. (2014). Retrieval (sometimes) enhances learning: performance pressure reduces the benefits of retrieval practice. Applied Cognitive Psychology, 28 , 597–606.

Hirshman, E. (2001). Elaboration in memory. In N. J. Smelser & P. B. Baltes (Eds.), International encyclopedia of the social & behavioral sciences (pp. 4369–4374). Oxford: Pergamon.

Chapter   Google Scholar  

Hobbiss, M. (2016). Make it meaningful! Elaboration [Blog post]. Retrieved from https://hobbolog.wordpress.com/2016/06/09/make-it-meaningful-elaboration/ . Accessed 25 Dec 2017.

Jones, F. (2016). Homework – is it really that useless? [Blog post]. Retrieved from http://www.learningscientists.org/blog/2016/4/5-1 . Accessed 25 Dec 2017.

Kaminski, J. A., & Sloutsky, V. M. (2013). Extraneous perceptual information interferes with children’s acquisition of mathematical knowledge. Journal of Educational Psychology, 105 (2), 351–363.

Kaminski, J. A., Sloutsky, V. M., & Heckler, A. F. (2008). The advantage of abstract examples in learning math. Science, 320 , 454–455.

Kang, S. H. (2016). Spaced repetition promotes efficient and effective learning policy implications for instruction. Policy Insights from the Behavioral and Brain Sciences, 3 , 12–19.

Kang, S. H. K., McDermott, K. B., & Roediger, H. L. (2007). Test format and corrective feedback modify the effects of testing on long-term retention. European Journal of Cognitive Psychology, 19 , 528–558.

Karpicke, J. D., & Aue, W. R. (2015). The testing effect is alive and well with complex materials. Educational Psychology Review, 27 , 317–326.

Karpicke, J. D., Blunt, J. R., Smith, M. A., & Karpicke, S. S. (2014). Retrieval-based learning: The need for guided retrieval in elementary school children. Journal of Applied Research in Memory and Cognition, 3 , 198–206.

Karpicke, J. D., Lehman, M., & Aue, W. R. (2014). Retrieval-based learning: an episodic context account. In B. H. Ross (Ed.), Psychology of Learning and Motivation (Vol. 61, pp. 237–284). San Diego, CA: Elsevier Academic Press.

Karpicke, J. D., Blunt, J. R., & Smith, M. A. (2016). Retrieval-based learning: positive effects of retrieval practice in elementary school children. Frontiers in Psychology, 7 .

Kavale, K. A., Hirshoren, A., & Forness, S. R. (1998). Meta-analytic validation of the Dunn and Dunn model of learning-style preferences: a critique of what was Dunn. Learning Disabilities Research & Practice, 13 , 75–80.

Khanna, M. M. (2015). Ungraded pop quizzes: test-enhanced learning without all the anxiety. Teaching of Psychology, 42 , 174–178.

Kirby, J. (2014). One scientific insight for curriculum design [Blog post]. Retrieved from https://pragmaticreform.wordpress.com/2014/05/05/scientificcurriculumdesign/ . Accessed 25 Dec 2017.

Kirschner, P. A. (2017). Stop propagating the learning styles myth. Computers & Education, 106 , 166–171.

Kirschner, P. A., & van Merriënboer, J. J. G. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48 , 169–183.

Knoll, A. R., Otani, H., Skeel, R. L., & Van Horn, K. R. (2017). Learning style, judgments of learning, and learning of verbal and visual information. British Journal of Psychology, 108 , 544-563.

Kornell, N., & Bjork, R. A. (2008). Learning concepts and categories is spacing the “enemy of induction”? Psychological Science, 19 , 585–592.

Kornell, N., & Finn, B. (2016). Self-regulated learning: an overview of theory and data. In J. Dunlosky & S. Tauber (Eds.), The Oxford Handbook of Metamemory (pp. 325–340). New York: Oxford University Press.

Kornell, N., Klein, P. J., & Rawson, K. A. (2015). Retrieval attempts enhance learning, but retrieval success (versus failure) does not matter. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41 , 283–294.

Kraemer, D. J. M., Rosenberg, L. M., & Thompson-Schill, S. L. (2009). The neural correlates of visual and verbal cognitive styles. Journal of Neuroscience, 29 , 3792–3798.

Article   PubMed   PubMed Central   Google Scholar  

Kraft, N. (2015). Spaced practice and repercussions for teaching. Retrieved from http://nathankraft.blogspot.com/2015/08/spaced-practice-and-repercussions-for.html . Accessed 25 Dec 2017.

Learning Scientists. (2016a). Weekly Digest #3: How teachers implement interleaving in their curriculum [Blog post]. Retrieved from http://www.learningscientists.org/blog/2016/3/28/weekly-digest-3 . Accessed 25 Dec 2017.

Learning Scientists. (2016b). Weekly Digest #13: how teachers implement retrieval in their classrooms [Blog post]. Retrieved from http://www.learningscientists.org/blog/2016/6/5/weekly-digest-13 . Accessed 25 Dec 2017.

Learning Scientists. (2016c). Weekly Digest #40: teachers’ implementation of principles from “Make It Stick” [Blog post]. Retrieved from http://www.learningscientists.org/blog/2016/12/18-1 . Accessed 25 Dec 2017.

Learning Scientists. (2017). Weekly Digest #54: is there an app for that? Studying 2.0 [Blog post]. Retrieved from http://www.learningscientists.org/blog/2017/4/9/weekly-digest-54 . Accessed 25 Dec 2017.

LeFevre, J.-A., & Dixon, P. (1986). Do written instructions need examples? Cognition and Instruction, 3 , 1–30.

Lew, K., Fukawa-Connelly, T., Mejí-Ramos, J. P., & Weber, K. (2016). Lectures in advanced mathematics: Why students might not understand what the mathematics professor is trying to convey. Journal of Research in Mathematics Education, 47 , 162–198.

Lindsey, R. V., Shroyer, J. D., Pashler, H., & Mozer, M. C. (2014). Improving students’ long-term knowledge retention through personalized review. Psychological Science, 25 , 639–647.

Lipko-Speed, A., Dunlosky, J., & Rawson, K. A. (2014). Does testing with feedback help grade-school children learn key concepts in science? Journal of Applied Research in Memory and Cognition, 3 , 171–176.

Lockhart, R. S., & Craik, F. I. (1990). Levels of processing: a retrospective commentary on a framework for memory research. Canadian Journal of Psychology, 44 , 87–112.

Lovell, O. (2017). How do we know what to put on the quiz? [Blog Post]. Retrieved from http://www.ollielovell.com/olliesclassroom/know-put-quiz/ . Accessed 25 Dec 2017.

Luehmann, A. L. (2008). Using blogging in support of teacher professional identity development: a case study. The Journal of the Learning Sciences, 17 , 287–337.

Madan, C. R., Glaholt, M. G., & Caplan, J. B. (2010). The influence of item properties on association-memory. Journal of Memory and Language, 63 , 46–63.

Madan, C. R., & Singhal, A. (2012a). Motor imagery and higher-level cognition: four hurdles before research can sprint forward. Cognitive Processing, 13 , 211–229.

Madan, C. R., & Singhal, A. (2012b). Encoding the world around us: motor-related processing influences verbal memory. Consciousness and Cognition, 21 , 1563–1570.

Madan, C. R., & Singhal, A. (2012c). Using actions to enhance memory: effects of enactment, gestures, and exercise on human memory. Frontiers in Psychology, 3 .

Madan, C. R., Chen, Y. Y., & Singhal, A. (2016). ERPs differentially reflect automatic and deliberate processing of the functional manipulability of objects. Frontiers in Human Neuroscience, 10 .

Mandler, G. (1979). Organization and repetition: organizational principles with special reference to rote learning. In L. G. Nilsson (Ed.), Perspectives on Memory Research (pp. 293–327). New York: Academic Press.

Marsh, E. J., Fazio, L. K., & Goswick, A. E. (2012). Memorial consequences of testing school-aged children. Memory, 20 , 899–906.

Mayer, R. E., & Gallini, J. K. (1990). When is an illustration worth ten thousand words? Journal of Educational Psychology, 82 , 715–726.

Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38 , 43–52.

McDaniel, M. A., & Donnelly, C. M. (1996). Learning with analogy and elaborative interrogation. Journal of Educational Psychology, 88 , 508–519.

McDaniel, M. A., Thomas, R. C., Agarwal, P. K., McDermott, K. B., & Roediger, H. L. (2013). Quizzing in middle-school science: successful transfer performance on classroom exams. Applied Cognitive Psychology, 27 , 360–372.

McDermott, K. B., Agarwal, P. K., D’Antonio, L., Roediger, H. L., & McDaniel, M. A. (2014). Both multiple-choice and short-answer quizzes enhance later exam performance in middle and high school classes. Journal of Experimental Psychology: Applied, 20 , 3–21.

McHugh, A. (2013). High-stakes tests: bad for students, teachers, and education in general [Blog post]. Retrieved from https://teacherbiz.wordpress.com/2013/07/01/high-stakes-tests-bad-for-students-teachers-and-education-in-general/ . Accessed 25 Dec 2017.

McNeill, N. M., Uttal, D. H., Jarvin, L., & Sternberg, R. J. (2009). Should you show me the money? Concrete objects both hurt and help performance on mathematics problems. Learning and Instruction, 19 , 171–184.

Meider, W. (1990). “A picture is worth a thousand words”: from advertising slogan to American proverb. Southern Folklore, 47 , 207–225.

Michaela Community School. (2014). Homework. Retrieved from http://mcsbrent.co.uk/homework-2/ . Accessed 25 Dec 2017.

Montefinese, M., Ambrosini, E., Fairfield, B., & Mammarella, N. (2013). The “subjective” pupil old/new effect: is the truth plain to see? International Journal of Psychophysiology, 89 , 48–56.

O’Neil, H. F., Chung, G. K., Kerr, D., Vendlinski, T. P., Buschang, R. E., & Mayer, R. E. (2014). Adding self-explanation prompts to an educational computer game. Computers In Human Behavior, 30 , 23–28.

Overoye, A. L., & Storm, B. C. (2015). Harnessing the power of uncertainty to enhance learning. Translational Issues in Psychological Science, 1 , 140–148.

Paivio, A. (1971). Imagery and verbal processes . New York: Holt, Rinehart and Winston.

Paivio, A. (1986). Mental representations: a dual coding approach . New York: Oxford University Press.

Paivio, A. (2007). Mind and its evolution: a dual coding theoretical approach . Mahwah: Erlbaum.

Paivio, A. (2013). Dual coding theory, word abstractness, and emotion: a critical review of Kousta et al. (2011). Journal of Experimental Psychology: General, 142 , 282–287.

Paivio, A., & Csapo, K. (1969). Concrete image and verbal memory codes. Journal of Experimental Psychology, 80 , 279–285.

Paivio, A., & Csapo, K. (1973). Picture superiority in free recall: imagery or dual coding? Cognitive Psychology, 5 , 176–206.

Paivio, A., Walsh, M., & Bons, T. (1994). Concreteness effects on memory: when and why? Journal of Experimental Psychology: Learning, Memory, and Cognition, 20 , 1196–1204.

Pashler, H., McDaniel, M., Rohrer, D., & Bjork, R. (2008). Learning styles: concepts and evidence. Psychological Science in the Public Interest, 9 , 105–119.

Pashler, H., Bain, P. M., Bottge, B. A., Graesser, A., Koedinger, K., McDaniel, M., & Metcalfe, J. (2007). Organizing instruction and study to improve student learning. IES practice guide. NCER 2007–2004. National Center for Education Research .

Patel, R., Liu, R., & Koedinger, K. (2016). When to block versus interleave practice? Evidence against teaching fraction addition before fraction multiplication. In Proceedings of the 38th Annual Meeting of the Cognitive Science Society, Philadelphia, PA .

Penfound, B. (2017). Journey to interleaved practice #2 [Blog Post]. Retrieved from https://fullstackcalculus.com/2017/02/03/journey-to-interleaved-practice-2/ . Accessed 25 Dec 2017.

Penfound, B. [BryanPenfound]. (2016). Does blocked practice/learning lessen cognitive load? Does interleaved practice/learning provide productive struggle? [Tweet]. Retrieved from https://twitter.com/BryanPenfound/status/808759362244087808 . Accessed 25 Dec 2017.

Peterson, D. J., & Mulligan, N. W. (2010). Enactment and retrieval. Memory & Cognition, 38 , 233–243.

Picciotto, H. (2009). Lagging homework [Blog post]. Retrieved from http://blog.mathedpage.org/2013/06/lagging-homework.html . Accessed 25 Dec 2017.

Pomerance, L., Greenberg, J., & Walsh, K. (2016). Learning about learning: what every teacher needs to know. Retrieved from http://www.nctq.org/dmsView/Learning_About_Learning_Report . Accessed 25 Dec 2017.

Postman, L. (1976). Methodology of human learning. In W. K. Estes (Ed.), Handbook of learning and cognitive processes (Vol. 3). Hillsdale: Erlbaum.

Pressley, M., McDaniel, M. A., Turnure, J. E., Wood, E., & Ahmad, M. (1987). Generation and precision of elaboration: effects on intentional and incidental learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 13 , 291–300.

Reed, S. K. (2008). Concrete examples must jibe with experience. Science, 322 , 1632–1633.

researchED. (2013). How it all began. Retrieved from http://www.researched.org.uk/about/our-story/ . Accessed 25 Dec 2017.

Ritchie, S. J., Della Sala, S., & McIntosh, R. D. (2013). Retrieval practice, with or without mind mapping, boosts fact learning in primary school children. PLoS One, 8 (11), e78976.

Rittle-Johnson, B. (2006). Promoting transfer: effects of self-explanation and direct instruction. Child Development, 77 , 1–15.

Roediger, H. L. (1985). Remembering Ebbinghaus. [Retrospective review of the book On Memory , by H. Ebbinghaus]. Contemporary Psychology, 30 , 519–523.

Roediger, H. L. (2013). Applying cognitive psychology to education translational educational science. Psychological Science in the Public Interest, 14 , 1–3.

Roediger, H. L., & Karpicke, J. D. (2006). The power of testing memory: basic research and implications for educational practice. Perspectives on Psychological Science, 1 , 181–210.

Roediger, H. L., Putnam, A. L., & Smith, M. A. (2011). Ten benefits of testing and their applications to educational practice. In J. Mester & B. Ross (Eds.), The psychology of learning and motivation: cognition in education (pp. 1–36). Oxford: Elsevier.

Roediger, H. L., Finn, B., & Weinstein, Y. (2012). Applications of cognitive science to education. In Della Sala, S., & Anderson, M. (Eds.), Neuroscience in education: the good, the bad, and the ugly . Oxford, UK: Oxford University Press.

Roelle, J., & Berthold, K. (2017). Effects of incorporating retrieval into learning tasks: the complexity of the tasks matters. Learning and Instruction, 49 , 142–156.

Rohrer, D. (2012). Interleaving helps students distinguish among similar concepts. Educational Psychology Review, 24(3), 355–367.

Rohrer, D., Dedrick, R. F., & Stershic, S. (2015). Interleaved practice improves mathematics learning. Journal of Educational Psychology, 107 , 900–908.

Rohrer, D., & Pashler, H. (2012). Learning styles: Where’s the evidence? Medical Education, 46 , 34–35.

Rohrer, D., & Taylor, K. (2007). The shuffling of mathematics problems improves learning. Instructional Science, 35 , 481–498.

Rose, N. (2014). Improving the effectiveness of homework [Blog post]. Retrieved from https://evidenceintopractice.wordpress.com/2014/03/20/improving-the-effectiveness-of-homework/ . Accessed 25 Dec 2017.

Sadoski, M. (2005). A dual coding view of vocabulary learning. Reading & Writing Quarterly, 21 , 221–238.

Saunders, K. (2016). It really is time we stopped talking about learning styles [Blog post]. Retrieved from http://martingsaunders.com/2016/10/it-really-is-time-we-stopped-talking-about-learning-styles/ . Accessed 25 Dec 2017.

Schwartz, D. (2007). If a picture is worth a thousand words, why are you reading this essay? Social Psychology Quarterly, 70 , 319–321.

Shumaker, H. (2016). Homework is wrecking our kids: the research is clear, let’s ban elementary homework. Salon. Retrieved from http://www.salon.com/2016/03/05/homework_is_wrecking_our_kids_the_research_is_clear_lets_ban_elementary_homework . Accessed 25 Dec 2017.

Smith, A. M., Floerke, V. A., & Thomas, A. K. (2016). Retrieval practice protects memory against acute stress. Science, 354 , 1046–1048.

Smith, M. A., Blunt, J. R., Whiffen, J. W., & Karpicke, J. D. (2016). Does providing prompts during retrieval practice improve learning? Applied Cognitive Psychology, 30 , 784–802.

Smith, M. A., & Karpicke, J. D. (2014). Retrieval practice with short-answer, multiple-choice, and hybrid formats. Memory, 22 , 784–802.

Smith, M. A., Roediger, H. L., & Karpicke, J. D. (2013). Covert retrieval practice benefits retention as much as overt retrieval practice. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39 , 1712–1725.

Son, J. Y., & Rivas, M. J. (2016). Designing clicker questions to stimulate transfer. Scholarship of Teaching and Learning in Psychology, 2 , 193–207.

Szpunar, K. K., Khan, N. Y., & Schacter, D. L. (2013). Interpolated memory tests reduce mind wandering and improve learning of online lectures. Proceedings of the National Academy of Sciences, 110 , 6313–6317.

Thomson, R., & Mehring, J. (2016). Better vocabulary study strategies for long-term learning. Kwansei Gakuin University Humanities Review, 20 , 133–141.

Trafton, J. G., & Reiser, B. J. (1993). Studying examples and solving problems: contributions to skill acquisition . Technical report, Naval HCI Research Lab, Washington, DC, USA.

Tran, R., Rohrer, D., & Pashler, H. (2015). Retrieval practice: the lack of transfer to deductive inferences. Psychonomic Bulletin & Review, 22 , 135–140.

Turner, K. [doc_kristy]. (2016a). My dual coding (in red) and some y8 work @AceThatTest they really enjoyed practising the technique [Tweet]. Retrieved from https://twitter.com/doc_kristy/status/807220355395977216 . Accessed 25 Dec 2017.

Turner, K. [doc_kristy]. (2016b). @FurtherEdagogy @doctorwhy their work is revision work, they already have the words on a different page, to compliment not replace [Tweet]. Retrieved from https://twitter.com/doc_kristy/status/807360265100599301 . Accessed 25 Dec 2017.

Valle, A., Regueiro, B., Núñez, J. C., Rodríguez, S., Piñeiro, I., & Rosário, P. (2016). Academic goals, student homework engagement, and academic achievement in elementary school. Frontiers in Psychology, 7 .

Van Gog, T., & Sweller, J. (2015). Not new, but nearly forgotten: the testing effect decreases or even disappears as the complexity of learning materials increases. Educational Psychology Review, 27 , 247–264.

Wammes, J. D., Meade, M. E., & Fernandes, M. A. (2016). The drawing effect: evidence for reliable and robust memory benefits in free recall. Quarterly Journal of Experimental Psychology, 69 , 1752–1776.

Weinstein, Y., Gilmore, A. W., Szpunar, K. K., & McDermott, K. B. (2014). The role of test expectancy in the build-up of proactive interference in long-term memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40 , 1039–1048.

Weinstein, Y., Nunes, L. D., & Karpicke, J. D. (2016). On the placement of practice questions during study. Journal of Experimental Psychology: Applied, 22 , 72–84.

Weinstein, Y., & Weinstein-Jones, F. (2017). Topic and quiz spacing spreadsheet: a planning tool for teachers [Blog Post]. Retrieved from http://www.learningscientists.org/blog/2017/5/11-1 . Accessed 25 Dec 2017.

Weinstein-Jones, F., & Weinstein, Y. (2017). Topic spacing spreadsheet for teachers [Excel macro]. Zenodo. http://doi.org/10.5281/zenodo.573764 . Accessed 25 Dec 2017.

Williams, D. [FurtherEdagogy]. (2016). @doctorwhy @doc_kristy word accompanying the visual? I’m unclear how removing words benefit? Would a flow chart better suit a scientific exp? [Tweet]. Retrieved from https://twitter.com/FurtherEdagogy/status/807356800509104128 . Accessed 25 Dec 2017.

Wood, B. (2017). And now for something a little bit different….[Blog post]. Retrieved from https://justateacherstandinginfrontofaclass.wordpress.com/2017/04/20/and-now-for-something-a-little-bit-different/ . Accessed 25 Dec 2017.

Wooldridge, C. L., Bugg, J. M., McDaniel, M. A., & Liu, Y. (2014). The testing effect with authentic educational materials: a cautionary note. Journal of Applied Research in Memory and Cognition, 3 , 214–221.

Young, C. (2016). Mini-tests. Retrieved from https://colleenyoung.wordpress.com/revision-activities/mini-tests/ . Accessed 25 Dec 2017.

Download references

Acknowledgements

Not applicable.

YW and MAS were partially supported by a grant from The IDEA Center.

Availability of data and materials

Author information, authors and affiliations.

Department of Psychology, University of Massachusetts Lowell, Lowell, MA, USA

Yana Weinstein

Department of Psychology, Boston College, Chestnut Hill, MA, USA

Christopher R. Madan

School of Psychology, University of Nottingham, Nottingham, UK

Department of Psychology, Rhode Island College, Providence, RI, USA

Megan A. Sumeracki

You can also search for this author in PubMed   Google Scholar

Contributions

YW took the lead on writing the “Spaced practice”, “Interleaving”, and “Elaboration” sections. CRM took the lead on writing the “Concrete examples” and “Dual coding” sections. MAS took the lead on writing the “Retrieval practice” section. All authors edited each others’ sections. All authors were involved in the conception and writing of the manuscript. All authors gave approval of the final version.

Corresponding author

Correspondence to Yana Weinstein .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

YW and MAS run a blog, “The Learning Scientists Blog”, which is cited in the tutorial review. The blog does not make money. Free resources on the strategies described in this tutorial review are provided on the blog. Occasionally, YW and MAS are invited by schools/school districts to present research findings from cognitive psychology applied to education.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Weinstein, Y., Madan, C.R. & Sumeracki, M.A. Teaching the science of learning. Cogn. Research 3 , 2 (2018). https://doi.org/10.1186/s41235-017-0087-y

Download citation

Received : 20 December 2016

Accepted : 02 December 2017

Published : 24 January 2018

DOI : https://doi.org/10.1186/s41235-017-0087-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Research landscape of adaptive learning in education: a bibliometric study on research publications from 2000 to 2022.

1. Introduction

2. literature review of adaptive learning, 3. methodology and materials, 3.1. bibliometric and research tools, 3.2. data retrieval, 3.3. data filtering, 4. performance analysis, 4.1. the development trend of research on adaptive learning research, 4.2. quantitative analysis of the authors, 4.3. quantitative analysis of the countries, 4.4. quantitative analysis of the journals, 5. co-occurrence analysis and evolution analysis, 5.1. co-occurrence analysis on keywords, 5.2. evolution analysis and research frontiers, 5.2.1. evolution analysis of the first stage (2000–2007), 5.2.2. evolution analysis of the second stage (2008–2013), 5.2.3. evolution analysis and frontier of the third stage (2013–2022), 6. conclusions, implications and future research, 6.1. conclusions, 6.2. findings and implications, 6.3. limitations and future research, author contributions, institutional review board statement, informed consent statement, data availability statement, acknowledgments, conflicts of interest.

• Capuano, N.; Caballé, S. Adaptive learning technologies. AI Mag. 2020 , 2 , 96–98. [ Google Scholar ] [ CrossRef ]
• Peng, H.; Ma, S.; Spector, J.M. Personalized adaptive learning: An emerging pedagogical approach enabled by a smart learning environment. Smart Learn. Environ. 2019 , 1 , 1–14. [ Google Scholar ] [ CrossRef ]
• Wang, S.; Christensen, C.; Cui, W.; Tong, R.; Yarnall, L.; Shear, L.; Feng, M. When adaptive learning is effective learning: Comparison of an adaptive learning system to teacher-led instruction. Intract. Learn. Environ. 2020 , 1–11. [ Google Scholar ] [ CrossRef ]
• Cavanagh, T.; Chen, B.; Lahcen, R.A.M.; Paradiso, J.R. Constructing a design framework and pedagogical approach for adaptive learning in higher education: A practitioner’s perspective. Int. Rev. Res. Open. Dis. 2020 , 1 , 173–197. [ Google Scholar ] [ CrossRef ]
• Campus Computing 2018: The 29th National Survey of Computing and Information Technology in American Higher Education. Available online: https://www.campuscomputing.net/content/2018/10/31/the-2018-campus-computing-survey (accessed on 26 December 2022).
• 2022 EDUCAUSE Horizon Report|Teaching and Learning Edition. Available online: https://library.educause.edu/-/media/files/library/2022/4/2022hrteachinglearning.pdf (accessed on 26 December 2022).
• Peng, T.; Zuo, W.; He, F. SVM based adaptive learning method for text classification from positive and unlabeled documents. Knowl. Inf. Syst. 2008 , 3 , 281–301. [ Google Scholar ] [ CrossRef ]
• Xie, H.; Chu, H.C.; Hwang, G.J.; Wang, C.C. Trends and development in technology-enhanced adaptive/personalized learning: A systematic review of journal publications from 2007 to 2017. Comput. Educ. 2019 , 140 , 103599. [ Google Scholar ] [ CrossRef ]
• Martin, F.; Chen, Y.; Moore, R.L.; Westine, C.D. Systematic review of adaptive learning research designs, context, strategies, and technologies from 2009 to 2018. ETRD-Educ. Technol. Res. 2020 , 4 , 1903–1929. [ Google Scholar ] [ CrossRef ]
• Villegas-Ch, W.; Roman-Cañizares, M.; Jaramillo-Alcázar, A.; Palacios-Pacheco, X. Data analysis as a tool for the application of adaptive learning in a university environment. Appl. Sci. 2020 , 20 , 7016. [ Google Scholar ] [ CrossRef ]
• Normadhi, N.B.A.; Shuib, L.; Nasir, H.N.M.; Bimba, A.; Idris, N.; Balakrishnan, V. Identification of personal traits in adaptive learning environment: Systematic literature review. Comput. Educ. 2019 , 130 , 168–190. [ Google Scholar ] [ CrossRef ]
• Li, F.; He, Y.; Xue, Q. Progress, challenges and countermeasures of adaptive learning. Educ. Technol. Soc. 2021 , 24 , 238–255. [ Google Scholar ]
• Hwang, G.J.; Tu, Y.F. Roles and research trends of artificial intelligence in mathematics education: A bibliometric mapping analysis and systematic review. Mathematics 2021 , 6 , 584. [ Google Scholar ] [ CrossRef ]
• Brusilovsky, P. Methods and techniques of adaptive hypermedia. In Adaptive Hypertext and Hypermedia ; Springer: Dordrecht, The Netherlands, 1998; Volume 1, pp. 1–43. [ Google Scholar ]
• UNESCO. International Forum on AI and the Futures of Education, Developing Competencies for the AI Era, 7–8 December 2020: Synthesis Report Synthesis Report ; UNESCO: Paris, France, 2020. [ Google Scholar ]
• Binet, A.; Simon, T. New methods for the diagnosis of the intellectual level of subnormals. (L’Année Psych., 1905, pp. 191–244). In The Development of Intelligence in Children (The Binet-Simon Scale) ; Binet, A.; Simon, T.; Kite, E.S., Translators; Williams & Wilkins Co.: Philadelphia, PA, USA, 1916; pp. 37–90. [ Google Scholar ]
• Pressey, S.L. A machine for automatic teaching of drill material. Sch. Soc. 1927 , 25 , 549–552. [ Google Scholar ]
• Yang, T.C.; Hwang, G.J.; Yang, S.J.H. Development of an adaptive learning system with multiple perspectives based on students’ learning styles and cognitive styles. Educ. Technol. Soc. 2013 , 4 , 185–200. [ Google Scholar ]
• Sarıyalçınkaya, A.D.; Karal, H.; Altinay, F.; Altinay, Z. Reflections on Adaptive Learning Analytics: Adaptive Learning Analytics. In Advancing the Power of Learning Analytics and Big Data in Education ; IGI Global: Hershey, PA, USA, 2021; Volume 1, pp. 61–84. [ Google Scholar ]
• Bernacki, M.L.; Greene, M.J.; Lobczowski, N.G. A systematic review of research on personalized learning: Personalized by whom, to what, how, and for what purpose (s)? Educ. Psychol. Rev. 2021 , 4 , 1675–1715. [ Google Scholar ] [ CrossRef ]
• Mo, C.Y.; Wang, C.; Dai, J.; Jin, P. Video Playback Speed Influence on Learning Effect from the Perspective of Personalized Adaptive Learning: A Study Based on Cognitive Load Theory. Front. Psychol. 2022 , 13 , 839982. [ Google Scholar ] [ CrossRef ]
• Diodato, V.P.; Gellatly, P. Dictionary of Bibliometrics , 1st ed.; Routledge: New York, NY, USA, 2013. [ Google Scholar ] [ CrossRef ]
• Donthu, N.; Kumar, S.; Pattnaik, D. Forty-five years of Journal of Business Research: A bibliometric analysis. J. Indian Bus. Res. 2020 , 109 , 1–14. [ Google Scholar ] [ CrossRef ]
• Pritchard, A. Statistical bibliography or bibliometrics? J. Doc. 1969 , 4 , 348–349. [ Google Scholar ]
• Verbeek, A.; Debackere, K.; Luwel, M.; Zimmermann, E. Measuring progress and evolution in science and technology-I: The multiple uses of bibliometric indicators. Int. J. Manag. Rev. 2002 , 2 , 179–211. [ Google Scholar ] [ CrossRef ]
• Yang, Y.; Wang, C.; Lai, M. Using bibliometric analysis to explore research trend of electronic word-of-mouth from 1999 to 2011. Int. J. Innov. Technol. 2012 , 3 , 337–342. [ Google Scholar ] [ CrossRef ]
• Wang, C.L.; Dai, J.; Xu, L.J. Big data and data mining in education: A bibliometrics study from 2010 to 2022. In Proceeding of the 2022 7th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA), Chengdu, China, 22–24 April 2022. [ Google Scholar ] [ CrossRef ]
• Van Eck, N.J.; Waltman, L. Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics 2010 , 2 , 523–538. [ Google Scholar ] [ CrossRef ]
• Chen, J.; Wang, C.; Tang, Y. Knowledge mapping of volunteer motivation: A bibliometric analysis and cross-cultural comparative study. Front. Psychol. 2022 , 13 , 883150. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ding, X.; Yang, Z. Knowledge mapping of platform research: A visual analysis using VOSviewer and CiteSpace. Electron. Commer. Res. 2020 , 22 , 1–23. [ Google Scholar ] [ CrossRef ]
• Popescu, D.V.; Dima, A.; Radu, E.; Dobrotă, E.M.; Dumitrache, V.M. Bibliometric Analysis of the Green Deal Policies in the Food Chain. Amfiteatru. Econ. 2022 , 24 , 410–428. [ Google Scholar ] [ CrossRef ]
• Dima, A.; Bugheanu, A.M.; Boghian, R.; Madsen, D.Ø. Mapping Knowledge Area Analysis in E-Learning Systems Based on Cloud Computing. Electronics 2023 , 12 , 62. [ Google Scholar ] [ CrossRef ]
• Chen, J.Y.; Liu, Y.D.; Dai, J.; Wang, C.L. Development and status of moral education research: Visual analysis based on knowledge graph. Front. Psychol. 2023 , 13 , 1079955. [ Google Scholar ] [ CrossRef ]
• Birkle, C.; Pendlebury, D.A.; Schnell, J.; Adams, J. Web of Science as a data source for research on scientific and scholarly activity. Quant. Sci. Stud. 2020 , 1 , 363–376. [ Google Scholar ] [ CrossRef ]
• Cobo, M.J.; Martínez, M.A.; Gutiérrez-Salcedo, M.; Fujita, H.; Herrera-Viedma, E. 25 years at knowledge-based systems: A bibliometric analysis. Knowl.-Based Syst. 2015 , 80 , 3–13. [ Google Scholar ] [ CrossRef ]
• Pranckute, R. Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World. Publications 2021 , 9 , 12. [ Google Scholar ] [ CrossRef ]
• Fernandez, G. Cognitive scaffolding for a web-based adaptive learning environment. In International Conference on Web-Based Learning (ICWL) ; Springer: Berlin/Heidelberg, Germany, 2003. [ Google Scholar ] [ CrossRef ]
• Nguyen, U.P.; Hallinger, P. Assessing the distinctive contributions of simulation & gaming to the literature, 1970–2019: A bibliometric review. Simulat. Gaming 2020 , 6 , 744–769. [ Google Scholar ] [ CrossRef ]
• Strotmann, A.; Zhao, D. Author name disambiguation: What difference does it make in author-based citation analysis? J. Am. Soc. Inf. Sci. Technol. 2012 , 9 , 1820–1833. [ Google Scholar ] [ CrossRef ]
• Han, H.; Wu, X.; Qiao, J. Nonlinear systems modeling based on self-organizing fuzzy-neural-network with adaptive computation algorithm. IEEE Trans. Cybern. 2014 , 4 , 554–564. [ Google Scholar ] [ CrossRef ]
• Qiao, J.; Li, F.; Han, H.; Li, W. Growing Echo-State Network with Multiple Subreservoirs. IEEE. Trans. Neur. Net. Learn. 2017 , 2 , 391–404. [ Google Scholar ] [ CrossRef ]
• Gu, K.; Zhou, J.; Qiao, J.F.; Zhai, G.; Lin, W.; Bovik, A.C. No-Reference Quality Assessment of Screen Content Pictures. IEEE. T. Image. Process. 2017 , 8 , 4005–4018. [ Google Scholar ] [ CrossRef ]
• Gu, K.; Tao, D.; Qiao, J.F.; Lin, W. Learning a No-Reference Quality Assessment Model of Enhanced Images with Big Data. IEEE. Trans. Neur. Net. Learn. 2018 , 4 , 1301–1313. [ Google Scholar ] [ CrossRef ]
• Han, H.G.; Chen, Q.L.; Qiao, J.F. An efficient self-organizing RBF neural network for water quality prediction. Neural. Netw. 2011 , 7 , 717–725. [ Google Scholar ] [ CrossRef ]
• Vamvoudakis, K.G.; Lewis, F.L. Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica. 2010 , 5 , 878–888. [ Google Scholar ] [ CrossRef ]
• Lewis, F.L.; Vrabie, D.; Vamvoudakis, K.G. Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers. IEEE. Contr. Syst. Mag. 2012 , 6 , 76–105. [ Google Scholar ] [ CrossRef ]
• Kiumarsi, B.; Vamvoudakis, K.G.; Modares, H.; Lewis, F.L. Optimal and autonomous control using reinforcement learning: A survey. IEEE Trans. Neur. Net. Learn. 2017 , 6 , 2042–2062. [ Google Scholar ] [ CrossRef ]
• Bhasin, S.; Kamalapurkar, R.; Johnson, M.; Vamvoudakis, K.G.; Lewis, F.L.; Dixon, W.E. A novel actor–critic–identifier architecture for approximate optimal control of uncertain nonlinear systems. Automatica 2013 , 1 , 82–92. [ Google Scholar ] [ CrossRef ]
• Wei, W.; Ge, J.; Xu, S.; Li, M.; Zhao, Z.; Li, X.; Zheng, J. Knowledge maps of disaster medicine in China based on co-word analysis. Disaster Med. Public 2019 , 3 , 405–409. [ Google Scholar ] [ CrossRef ]
• Bodily, R.; Leary, H.; West, R.E. Research trends in instructional design and technology journals. Br. J. Educ. Technol. 2019 , 1 , 64–79. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• De Melo, F.R.; Flôres, E.L.; De Carvalho, S.D.; De Teixeira, R.A.G.; Loja, L.F.B.; de Sousa Gomide, R. Computational organization of didactic contents for personalized virtual learning environments. Comput. Educ. 2014 , 79 , 126–137. [ Google Scholar ] [ CrossRef ]
• Holmes, M.; Latham, A.; Crockett, K.; O’Shea, J.D. Near real-time comprehension classification with artificial neural networks: Decoding e-learner non-verbal behavior. IEEE Trans. Learn. Technol. 2017 , 1 , 5–12. [ Google Scholar ] [ CrossRef ]
• Dutt, S.; Ahuja, N.J.; Kumar, M. An intelligent tutoring system architecture based on fuzzy neural network (FNN) for special education of learning disabled learners. Educ. Inf. Technol. 2022 , 2 , 2613–2633. [ Google Scholar ] [ CrossRef ]
• Chen, X.; Zou, D.; Xie, H.; Chen, G.; Lin, J.; Cheng, G. Exploring contributors, collaborations, and research topics in educational technology: A joint analysis of mainstream conferences. Educ. Inf. Technol. 2022 , 1–36. [ Google Scholar ] [ CrossRef ]
• Vanitha, V.; Krishnan, P.; Elakkiya, R. Collaborative optimization algorithm for learning path construction in E-learning. Comput. Electr. Eng. 2019 , 77 , 325–338. [ Google Scholar ] [ CrossRef ]
• Zhou, Y.; Huang, C.; Hu, Q.; Zhu, J.; Tang, Y. Personalized learning full-path recommendation model based on LSTM neural networks. Inf. Sci. 2018 , 444 , 135–152. [ Google Scholar ] [ CrossRef ]
• Dwivedi, P.; Kant, V.; Bharadwaj, K.K. Learning path recommendation based on modifed variable length genetic algorithm. Educ. Inf. Technol. 2018 , 2 , 819–836. [ Google Scholar ] [ CrossRef ]
• Wang, X.; Xing, W.; Lafey, J.M. Autistic youth in 3D game-based collaborative virtual learning: Associating avatar interaction patterns with embodied social presence. Brit. J. Educ. Technol. 2018 , 4 , 742–760. [ Google Scholar ] [ CrossRef ]
• Chen, H.; Park, H.W.; Breazeal, C. Teaching and learning with children: Impact of reciprocal peer learning with a social robot on children’s learning and emotive engagement. Comput. Educ. 2020 , 150 , 103836. [ Google Scholar ] [ CrossRef ]
• Zhang, H.; Huang, T.; Liu, S.; Yin, H.; Li, J.; Yang, H.; Xia, Y. A learning style classification approach based on deep belief network for large-scale online education. J. Cloud. Comput. 2020 , 1 , 1–17. [ Google Scholar ] [ CrossRef ]
• Bang, H.J.; Li, L.; Flynn, K. Efficacy of an Adaptive Game-Based Math Learning App to Support Personalized Learning and Improve Early Elementary School Students’ Learning. Early. Child. Educ. J. 2022 , 1–16. [ Google Scholar ] [ CrossRef ]
• Kärki, T.; McMullen, J.; Lehtinen, E. Improving rational number knowledge using the NanoRoboMath digital game. Educ. Stud. Math. 2022 , 1 , 101–123. [ Google Scholar ] [ CrossRef ]
• Faber, T.J.; Dankbaar, M.E.; Kickert, R.; van den Broek, W.W.; van Merriënboer, J.J. Identifying indicators to guide adaptive scaffolding in games. Learn. Instr. 2022 , 83 , 101666. [ Google Scholar ] [ CrossRef ]
• Zeng, L.; Wang, Y. Independent and innovative learning of gymnastics teaching based on computer interaction. Int. J. Elec. Eng. Educ. 2021 , 0020720920984325. [ Google Scholar ] [ CrossRef ]
• Daghestani, L.F.; Ibrahim, L.F.; Al-Towirgi, R.S.; Salman, H.A. Adapting gamified learning systems using educational data mining techniques. Comput. Appl. Eng. Educ. 2020 , 3 , 568–589. [ Google Scholar ] [ CrossRef ]
• Nilashi, M.; Abumalloh, R.A.; Zibarzani, M.; Samad, S.; Zogaan, W.A.; Ismail, M.Y.; Akib, N.A.M. What Factors Influence Students Satisfaction in Massive Open Online Courses? Findings from User-Generated Content Using Educational Data Mining. Educ. Inf. Technol. 2022 , 7 , 1–35. [ Google Scholar ] [ CrossRef ]
• Lo, J.J.; Shu, P.C. Identification of learning styles online by observing learners’ browsing behaviour through a neural network. Brit. J. Educ. Technol. 2005 , 1 , 43–55. [ Google Scholar ] [ CrossRef ]
• Mavroudi, A.; Giannakos, M.; Krogstie, J. Supporting adaptive learning pathways through the use of learning analytics: Developments, challenges and future opportunities. Intract. Learn. Environ. 2018 , 2 , 206–220. [ Google Scholar ] [ CrossRef ]
• Peng, P.; Fu, W. A pattern recognition method of personalized adaptive learning in online education. Mobile. Netw. Appl. 2022 , 3 , 1–13. [ Google Scholar ] [ CrossRef ]
• Roach, M.; Blackmore, P.; Dempster, J.A. Supporting high-level learning through research-based methods: A framework for course development. Innov. Educ. Teach. Int. 2001 , 4 , 369–382. [ Google Scholar ] [ CrossRef ]
• Nedungadi, P.; Raman, R. A new approach to personalization: Integrating e-learning and m-learning. ERTD-Educ. Tech. Res. 2012 , 4 , 659–678. [ Google Scholar ] [ CrossRef ]
• Tseng, S.S.; Su, J.M.; Hwang, G.J.; Hwang, G.H.; Tsai, C.C.; Tsai, C.J. An object-oriented course framework for developing adaptive learning systems. Educ. Technol. Soc. 2008 , 2 , 171–191. [ Google Scholar ]
• Yaghmaie, M.; Bahreininejad, A. A context-aware adaptive learning system using agents. Expert. Syst. Appl. 2011 , 4 , 3280–3286. [ Google Scholar ] [ CrossRef ]
• Wu, L.J.; Chang, K.E. Effect of embedding a cognitive diagnosis into the adaptive dynamic assessment of spatial geometry learning. Intract. Learn. Environ. 2020 , 1–18. [ Google Scholar ] [ CrossRef ]
• Zhang, J.; Yuan, C.; Wang, C.; Zeng, W.; Dai, S.L. Intelligent adaptive learning and control for discrete-time nonlinear uncertain systems in multiple environments. Neurocomputing 2021 , 462 , 31–45. [ Google Scholar ] [ CrossRef ]
• Luo, Z. Using eye-tracking technology to identify learning styles: Behaviour patterns and identification accuracy. Educ. Inf. Technol. 2021 , 4 , 4457–4485. [ Google Scholar ] [ CrossRef ]
• Reinstein, I.; Hill, J.; Cook, D.A.; Lineberry, M.; Pusic, M.V. Multi-level longitudinal learning curve regression models integrated with item difficulty metrics for deliberate practice of visual diagnosis: Groundwork for adaptive learning. Adv. Health. Sci. Educ. 2021 , 3 , 881–912. [ Google Scholar ] [ CrossRef ]
• Wan, H.; Yu, S. A recommendation system based on an adaptive learning cognitive map model and its effects. Intract. Learn. Environ. 2020 , 1–19. [ Google Scholar ] [ CrossRef ]
• Zourmpakis, A.-I.; Papadakis, S.; Kalogiannakis, M. Education of Preschool and Elementary Teachers on the Use of Adaptive Gamification in Science Education. Int. J. Technol. Enhanc. Learn. 2022 , 14 , 1–16. [ Google Scholar ] [ CrossRef ]
• Beyhan, S.; Alci, M. Extended fuzzy function model with stable learning methods for online system identification. Int. J. Adapt. Control. 2011 , 2 , 168–182. [ Google Scholar ] [ CrossRef ]
• Yang, Y.T.C.; Gamble, J.H.; Hung, Y.W.; Lin, T.Y. An online adaptive learning environment for critical-thinking-infused English literacy instruction. Brit. J. Educ. Technol. 2014 , 4 , 723–747. [ Google Scholar ] [ CrossRef ]
• Sridharan, S.; Saravanan, D.; Srinivasan, A.K.; Murugan, B. Adaptive learning management expert system with evolving knowledge base and enhanced learnability. Educ. Inf. Technol. 2021 , 5 , 5895–5916. [ Google Scholar ] [ CrossRef ]
• González-Castro, N.; Muñoz-Merino, P.J.; Alario-Hoyos, C.; Kloos, C.D. Adaptive learning module for a conversational agent to support MOOC learners. Australas. J. Educ. Technol. 2021 , 2 , 24–44. [ Google Scholar ] [ CrossRef ]
• Agand, P.; Shoorehdeli, M.A.; Khaki-Sedigh, A. Adaptive recurrent neural network with Lyapunov stability learning rules for robot dynamic terms identification. Eng. Appl. Artif. Intel. 2017 , 65 , 1–11. [ Google Scholar ] [ CrossRef ]
• Hassan, M.A.; Habiba, U.; Majeed, F.; Shoaib, M. Adaptive gamification in e-learning based on students’ learning styles. Intract. Learn. Environ. 2021 , 4 , 545–565. [ Google Scholar ] [ CrossRef ]
• Binh, H.T.; Trung, N.Q. Responsive student model in an intelligent tutoring system and its evaluation. Educ. Inf. Technol. 2021 , 4 , 4969–4991. [ Google Scholar ] [ CrossRef ]
• Caballé, S.; Mora, N.; Feidakis, M.; Gañán, D.; Conesa, J.; Daradoumis, T.; Prieto, J. CC–LR: Providing interactive, challenging and attractive Collaborative Complex Learning Resources. J. Comput. Assist. Learn. 2014 , 1 , 51–67. [ Google Scholar ] [ CrossRef ]
• Chu, H.C.; Chen, J.M.; Kuo, F.R.; Yang, S.M. Development of an adaptive game-based diagnostic and remedial learning system based on the concept-effect model for improving learning achievements in mathematics. Educ. Technol. Soc. 2021 , 4 , 36–53. [ Google Scholar ]
• Azevedo, R.; Moos, D.C.; Johnson, A.M.; Chauncey, A.D. Measuring cognitive and metacognitive regulatory processes during hypermedia learning: Issues and challenges. Educ. Psychol. 2010 , 4 , 210–223. [ Google Scholar ] [ CrossRef ]
• Lin, Y.T.; Wu, C.C.; Hou, T.Y.; Lin, Y.C.; Yang, F.Y.; Chang, C.H. Tracking students’ cognitive processes during program debugging—An eye-movement approach. IEEE Trans. Educ. 2015 , 3 , 175–186. [ Google Scholar ] [ CrossRef ]
• Tempelaar, D. Supporting the less-adaptive student: The role of learning analytics, formative assessment and blended learning. Assess. Eval. High. Educ. 2020 , 4 , 579–593. [ Google Scholar ] [ CrossRef ]
• Narciss, S.; Sosnovsky, S.; Schnaubert, L.; Andrès, E.; Eichelmann, A.; Goguadze, G.; Melis, E. Exploring feedback and student characteristics relevant for personalizing feedback strategies. Comput. Educ. 2014 , 71 , 56–76. [ Google Scholar ] [ CrossRef ]
• Sharma, K.; Papamitsiou, Z.; Giannakos, M. Building pipelines for educational data using AI and multimodal analytics: A “grey-box” approach. Br. J. Educ. Technol. 2019 , 6 , 3004–3031. [ Google Scholar ] [ CrossRef ]
• Gandomkar, R.; Yazdani, K.; Fata, L.; Mehrdad, R.; Mirzazadeh, A.; Jalili, M.; Sandars, J. Using multiple self-regulated learning measures to understand medical students’ biomedical science learning. Med. Educ. 2020 , 8 , 727–737. [ Google Scholar ] [ CrossRef ]
• Schumacher, C.; Ifenthaler, D. Investigating prompts for supporting students’ self-regulation–A remaining challenge for learning analytics approaches? Internet High. Educ. 2021 , 49 , 100791. [ Google Scholar ] [ CrossRef ]
• Wang, Y.; Liu, L. Research on the Characteristic Model of Learners in Modern Distance Music Classroom Based on Big Data. Sci. Program 2022 , 2022 , 4684461. [ Google Scholar ] [ CrossRef ]
• Adnan, M.; Alarood, A.A.S.; Uddin, M.I.; ur Rehman, I. Utilizing grid search cross-validation with adaptive boosting for augmenting performance of machine learning models. Peerj. Comput. Sci. 2022 , 8 , e803. [ Google Scholar ] [ CrossRef ]
• Liu, H.I.; Yang, M.N. QoL guaranteed adaptation and personalization in e-learning systems. IEEE Trans. Educ. 2005 , 4 , 676–687. [ Google Scholar ] [ CrossRef ]
• Wang, F.H. A fuzzy neural network for item sequencing in personalized cognitive scaffolding with adaptive formative assessment. Expert. Syst. Appl. 2004 , 1 , 11–25. [ Google Scholar ] [ CrossRef ]
• Karampiperis, P.; Sampson, D. Adaptive learning resources sequencing in educational hypermedia systems. Educ. Technol. Soc. 2005 , 4 , 128–147. [ Google Scholar ]
• Santos, J.M.; Anido, L.; Llamas, M.; Álvarez, L.M.; Mikic, F.A. Applying computational science techniques to support adaptive learning. In Proceedings of the International Conference on Computational Science (ICCS), St. Petersburg, Russia, 2–4 June 2003; Springer: Berlin/Heidelberg, Germany, 2003. [ Google Scholar ] [ CrossRef ]
• Tsai, C.J.; Tseng, S.S.; Lin, C.Y. A two-phase fuzzy mining and learning algorithm for adaptive learning environment. In Proceedings of the International Conference on Computational Science (ICCS), San Francisco, CA, USA, 28–30 May 2001; Springer: Berlin/Heidelberg, Germany, 2001. [ Google Scholar ] [ CrossRef ]
• Tseng, J.C.; Chu, H.C.; Hwang, G.J.; Tsai, C.C. Development of an adaptive learning system with two sources of personalization information. Comput. Educ. 2008 , 2 , 776–786. [ Google Scholar ] [ CrossRef ]
• Lin, C.F.; Yeh, Y.C.; Hung, Y.H.; Chang, R.I. Data mining for providing a personalized learning path in creativity: An application of decision trees. Comput. Educ. 2013 , 68 , 199–210. [ Google Scholar ] [ CrossRef ]
• Ebibi, M.; Fetaji, B.; Fetaji, M. Expert Based Learning (EXBL) Methodology for Developing Mobile Expert Learning Knowledge Management Software System. Tech. Technol. Educ. Manag. 2012 , 2 , 864–874. [ Google Scholar ]
• Chang, T.Y.; Ke, Y.R. A personalized e-course composition based on a genetic algorithm with forcing legality in an adaptive learning system. J. Netw. Comput. Appl. 2013 , 1 , 533–542. [ Google Scholar ] [ CrossRef ]
• Coello, C.C.; Lechuga, M.S. MOPSO: A proposal for multiple objective particle swarm optimization. In Proceedings of the 2002 IEEE Congress on Evolutionary Computation, Honolulu, HI, USA, 12–17 May 2002; Volume 2, pp. 1051–1056. [ Google Scholar ] [ CrossRef ]
• Martins, A.F.; Machado, M.; Bernardino, H.S.; de Souza, J.F. A comparative analysis of metaheuristics applied to adaptive curriculum sequencing. Soft Comput. 2021 , 16 , 11019–11034. [ Google Scholar ] [ CrossRef ]
• LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature 2015 , 7553 , 436–444. [ Google Scholar ] [ CrossRef ]
• Zhang, L.; Amat, A.Z.; Zhao, H.; Swanson, A.; Weitlauf, A.; Warren, Z.; Sarkar, N. Design of an intelligent agent to measure collaboration and verbal-communication skills of children with autism spectrum disorder in collaborative puzzle games. IEEE Trans. Learn. Technol. 2020 , 3 , 338–352. [ Google Scholar ] [ CrossRef ]
• Dhakshinamoorthy, A.; Dhakshinamoorthy, K. KLSAS—An adaptive dynamic learning environment based on knowledge level and learning style. Comput. Appl. Eng. Educ. 2019 , 2 , 319–331. [ Google Scholar ] [ CrossRef ]
• Pavlik, P.I.; Eglington, L.G.; Harrell-Williams, L.M. Logistic knowledge tracing: A constrained framework for learner modeling. IEEE Trans. Learn. Technol. 2021 , 5 , 624–639. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Jing, Y.; Zhao, L.; Zhu, K.; Wang, H.; Wang, C.; Xia, Q. Research Landscape of Adaptive Learning in Education: A Bibliometric Study on Research Publications from 2000 to 2022. Sustainability 2023 , 15 , 3115. https://doi.org/10.3390/su15043115

Jing Y, Zhao L, Zhu K, Wang H, Wang C, Xia Q. Research Landscape of Adaptive Learning in Education: A Bibliometric Study on Research Publications from 2000 to 2022. Sustainability . 2023; 15(4):3115. https://doi.org/10.3390/su15043115

Jing, Yuhui, Leying Zhao, Keke Zhu, Haoming Wang, Chengliang Wang, and Qi Xia. 2023. "Research Landscape of Adaptive Learning in Education: A Bibliometric Study on Research Publications from 2000 to 2022" Sustainability 15, no. 4: 3115. https://doi.org/10.3390/su15043115

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

• Research article
• Open access
• Published: 06 February 2017

Blended learning effectiveness: the relationship between student characteristics, design features and outcomes

• Mugenyi Justice Kintu   ORCID: orcid.org/0000-0002-4500-1168 1 , 2 ,
• Chang Zhu 2 &
• Edmond Kagambe 1  

International Journal of Educational Technology in Higher Education volume  14 , Article number:  7 ( 2017 ) Cite this article

779k Accesses

247 Citations

36 Altmetric

Metrics details

This paper investigates the effectiveness of a blended learning environment through analyzing the relationship between student characteristics/background, design features and learning outcomes. It is aimed at determining the significant predictors of blended learning effectiveness taking student characteristics/background and design features as independent variables and learning outcomes as dependent variables. A survey was administered to 238 respondents to gather data on student characteristics/background, design features and learning outcomes. The final semester evaluation results were used as a measure for performance as an outcome. We applied the online self regulatory learning questionnaire for data on learner self regulation, the intrinsic motivation inventory for data on intrinsic motivation and other self-developed instruments for measuring the other constructs. Multiple regression analysis results showed that blended learning design features (technology quality, online tools and face-to-face support) and student characteristics (attitudes and self-regulation) predicted student satisfaction as an outcome. The results indicate that some of the student characteristics/backgrounds and design features are significant predictors for student learning outcomes in blended learning.

Introduction

The teaching and learning environment is embracing a number of innovations and some of these involve the use of technology through blended learning. This innovative pedagogical approach has been embraced rapidly though it goes through a process. The introduction of blended learning (combination of face-to-face and online teaching and learning) initiatives is part of these innovations but its uptake, especially in the developing world faces challenges for it to be an effective innovation in teaching and learning. Blended learning effectiveness has quite a number of underlying factors that pose challenges. One big challenge is about how users can successfully use the technology and ensuring participants’ commitment given the individual learner characteristics and encounters with technology (Hofmann, 2014 ). Hofmann adds that users getting into difficulties with technology may result into abandoning the learning and eventual failure of technological applications. In a report by Oxford Group ( 2013 ), some learners (16%) had negative attitudes to blended learning while 26% were concerned that learners would not complete study in blended learning. Learners are important partners in any learning process and therefore, their backgrounds and characteristics affect their ability to effectively carry on with learning and being in blended learning, the design tools to be used may impinge on the effectiveness in their learning.

This study tackles blended learning effectiveness which has been investigated in previous studies considering grades, course completion, retention and graduation rates but no studies regarding effectiveness in view of learner characteristics/background, design features and outcomes have been done in the Ugandan university context. No studies have also been done on how the characteristics of learners and design features are predictors of outcomes in the context of a planning evaluation research (Guskey, 2000 ) to establish the effectiveness of blended learning. Guskey ( 2000 ) noted that planning evaluation fits in well since it occurs before the implementation of any innovation as well as allowing planners to determine the needs, considering participant characteristics, analyzing contextual matters and gathering baseline information. This study is done in the context of a plan to undertake innovative pedagogy involving use of a learning management system (moodle) for the first time in teaching and learning in a Ugandan university. The learner characteristics/backgrounds being investigated for blended learning effectiveness include self-regulation, computer competence, workload management, social and family support, attitude to blended learning, gender and age. We investigate the blended learning design features of learner interactions, face-to-face support, learning management system tools and technology quality while the outcomes considered include satisfaction, performance, intrinsic motivation and knowledge construction. Establishing the significant predictors of outcomes in blended learning will help to inform planners of such learning environments in order to put in place necessary groundwork preparations for designing blended learning as an innovative pedagogical approach.

Kenney and Newcombe ( 2011 ) did their comparison to establish effectiveness in view of grades and found that blended learning had higher average score than the non-blended learning environment. Garrison and Kanuka ( 2004 ) examined the transformative potential of blended learning and reported an increase in course completion rates, improved retention and increased student satisfaction. Comparisons between blended learning environments have been done to establish the disparity between academic achievement, grade dispersions and gender performance differences and no significant differences were found between the groups (Demirkol & Kazu, 2014 ).

However, blended learning effectiveness may be dependent on many other factors and among them student characteristics, design features and learning outcomes. Research shows that the failure of learners to continue their online education in some cases has been due to family support or increased workload leading to learner dropout (Park & Choi, 2009 ) as well as little time for study. Additionally, it is dependent on learner interactions with instructors since failure to continue with online learning is attributed to this. In Greer, Hudson & Paugh’s study as cited in Park and Choi ( 2009 ), family and peer support for learners is important for success in online and face-to-face learning. Support is needed for learners from all areas in web-based courses and this may be from family, friends, co-workers as well as peers in class. Greer, Hudson and Paugh further noted that peer encouragement assisted new learners in computer use and applications. The authors also show that learners need time budgeting, appropriate technology tools and support from friends and family in web-based courses. Peer support is required by learners who have no or little knowledge of technology, especially computers, to help them overcome fears. Park and Choi, ( 2009 ) showed that organizational support significantly predicts learners’ stay and success in online courses because employers at times are willing to reduce learners’ workload during study as well as supervisors showing that they are interested in job-related learning for employees to advance and improve their skills.

The study by Kintu and Zhu ( 2016 ) investigated the possibility of blended learning in a Ugandan University and examined whether student characteristics (such as self-regulation, attitudes towards blended learning, computer competence) and student background (such as family support, social support and management of workload) were significant factors in learner outcomes (such as motivation, satisfaction, knowledge construction and performance). The characteristics and background factors were studied along with blended learning design features such as technology quality, learner interactions, and Moodle with its tools and resources. The findings from that study indicated that learner attitudes towards blended learning were significant factors to learner satisfaction and motivation while workload management was a significant factor to learner satisfaction and knowledge construction. Among the blended learning design features, only learner interaction was a significant factor to learner satisfaction and knowledge construction.

The focus of the present study is on examining the effectiveness of blended learning taking into consideration learner characteristics/background, blended learning design elements and learning outcomes and how the former are significant predictors of blended learning effectiveness.

Studies like that of Morris and Lim ( 2009 ) have investigated learner and instructional factors influencing learning outcomes in blended learning. They however do not deal with such variables in the contexts of blended learning design as an aspect of innovative pedagogy involving the use of technology in education. Apart from the learner variables such as gender, age, experience, study time as tackled before, this study considers social and background aspects of the learners such as family and social support, self-regulation, attitudes towards blended learning and management of workload to find out their relationship to blended learning effectiveness. Identifying the various types of learner variables with regard to their relationship to blended learning effectiveness is important in this study as we embark on innovative pedagogy with technology in teaching and learning.

Literature review

This review presents research about blended learning effectiveness from the perspective of learner characteristics/background, design features and learning outcomes. It also gives the factors that are considered to be significant for blended learning effectiveness. The selected elements are as a result of the researcher’s experiences at a Ugandan university where student learning faces challenges with regard to learner characteristics and blended learning features in adopting the use of technology in teaching and learning. We have made use of Loukis, Georgiou, and Pazalo ( 2007 ) value flow model for evaluating an e-learning and blended learning service specifically considering the effectiveness evaluation layer. This evaluates the extent of an e-learning system usage and the educational effectiveness. In addition, studies by Leidner, Jarvenpaa, Dillon and Gunawardena as cited in Selim ( 2007 ) have noted three main factors that affect e-learning and blended learning effectiveness as instructor characteristics, technology and student characteristics. Heinich, Molenda, Russell, and Smaldino ( 2001 ) showed the need for examining learner characteristics for effective instructional technology use and showed that user characteristics do impact on behavioral intention to use technology. Research has dealt with learner characteristics that contribute to learner performance outcomes. They have dealt with emotional intelligence, resilience, personality type and success in an online learning context (Berenson, Boyles, & Weaver, 2008 ). Dealing with the characteristics identified in this study will give another dimension, especially for blended learning in learning environment designs and add to specific debate on learning using technology. Lin and Vassar, ( 2009 ) indicated that learner success is dependent on ability to cope with technical difficulty as well as technical skills in computer operations and internet navigation. This justifies our approach in dealing with the design features of blended learning in this study.

Learner characteristics/background and blended learning effectiveness

Studies indicate that student characteristics such as gender play significant roles in academic achievement (Oxford Group, 2013 ), but no study examines performance of male and female as an important factor in blended learning effectiveness. It has again been noted that the success of e- and blended learning is highly dependent on experience in internet and computer applications (Picciano & Seaman, 2007 ). Rigorous discovery of such competences can finally lead to a confirmation of high possibilities of establishing blended learning. Research agrees that the success of e-learning and blended learning can largely depend on students as well as teachers gaining confidence and capability to participate in blended learning (Hadad, 2007 ). Shraim and Khlaif ( 2010 ) note in their research that 75% of students and 72% of teachers were lacking in skills to utilize ICT based learning components due to insufficient skills and experience in computer and internet applications and this may lead to failure in e-learning and blended learning. It is therefore pertinent that since the use of blended learning applies high usage of computers, computer competence is necessary (Abubakar & Adetimirin, 2015 ) to avoid failure in applying technology in education for learning effectiveness. Rovai, ( 2003 ) noted that learners’ computer literacy and time management are crucial in distance learning contexts and concluded that such factors are meaningful in online classes. This is supported by Selim ( 2007 ) that learners need to posses time management skills and computer skills necessary for effectiveness in e- learning and blended learning. Self-regulatory skills of time management lead to better performance and learners’ ability to structure the physical learning environment leads to efficiency in e-learning and blended learning environments. Learners need to seek helpful assistance from peers and teachers through chats, email and face-to-face meetings for effectiveness (Lynch & Dembo, 2004 ). Factors such as learners’ hours of employment and family responsibilities are known to impede learners’ process of learning, blended learning inclusive (Cohen, Stage, Hammack, & Marcus, 2012 ). It was also noted that a common factor in failure and learner drop-out is the time conflict which is compounded by issues of family , employment status as well as management support (Packham, Jones, Miller, & Thomas, 2004 ). A study by Thompson ( 2004 ) shows that work, family, insufficient time and study load made learners withdraw from online courses.

Learner attitudes to blended learning can result in its effectiveness and these shape behavioral intentions which usually lead to persistence in a learning environment, blended inclusive. Selim, ( 2007 ) noted that the learners’ attitude towards e-learning and blended learning are success factors for these learning environments. Learner performance by age and gender in e-learning and blended learning has been found to indicate no significant differences between male and female learners and different age groups (i.e. young, middle-aged and old above 45 years) (Coldwell, Craig, Paterson, & Mustard, 2008 ). This implies that the potential for blended learning to be effective exists and is unhampered by gender or age differences.

Blended learning design features

The design features under study here include interactions, technology with its quality, face-to-face support and learning management system tools and resources.

Research shows that absence of learner interaction causes failure and eventual drop-out in online courses (Willging & Johnson, 2009 ) and the lack of learner connectedness was noted as an internal factor leading to learner drop-out in online courses (Zielinski, 2000 ). It was also noted that learners may not continue in e- and blended learning if they are unable to make friends thereby being disconnected and developing feelings of isolation during their blended learning experiences (Willging & Johnson, 2009). Learners’ Interactions with teachers and peers can make blended learning effective as its absence makes learners withdraw (Astleitner, 2000 ). Loukis, Georgious and Pazalo (2007) noted that learners’ measuring of a system’s quality, reliability and ease of use leads to learning efficiency and can be so in blended learning. Learner success in blended learning may substantially be affected by system functionality (Pituch & Lee, 2006 ) and may lead to failure of such learning initiatives (Shrain, 2012 ). It is therefore important to examine technology quality for ensuring learning effectiveness in blended learning. Tselios, Daskalakis, and Papadopoulou ( 2011 ) investigated learner perceptions after a learning management system use and found out that the actual system use determines the usefulness among users. It is again noted that a system with poor response time cannot be taken to be useful for e-learning and blended learning especially in cases of limited bandwidth (Anderson, 2004 ). In this study, we investigate the use of Moodle and its tools as a function of potential effectiveness of blended learning.

The quality of learning management system content for learners can be a predictor of good performance in e-and blended learning environments and can lead to learner satisfaction. On the whole, poor quality technology yields no satisfaction by users and therefore the quality of technology significantly affects satisfaction (Piccoli, Ahmad, & Ives, 2001 ). Continued navigation through a learning management system increases use and is an indicator of success in blended learning (Delone & McLean, 2003 ). The efficient use of learning management system and its tools improves learning outcomes in e-learning and blended learning environments.

It is noted that learner satisfaction with a learning management system can be an antecedent factor for blended learning effectiveness. Goyal and Tambe ( 2015 ) noted that learners showed an appreciation to Moodle’s contribution in their learning. They showed positivity with it as it improved their understanding of course material (Ahmad & Al-Khanjari, 2011 ). The study by Goyal and Tambe ( 2015 ) used descriptive statistics to indicate improved learning by use of uploaded syllabus and session plans on Moodle. Improved learning is also noted through sharing study material, submitting assignments and using the calendar. Learners in the study found Moodle to be an effective educational tool.

In blended learning set ups, face-to-face experiences form part of the blend and learner positive attitudes to such sessions could mean blended learning effectiveness. A study by Marriot, Marriot, and Selwyn ( 2004 ) showed learners expressing their preference for face-to-face due to its facilitation of social interaction and communication skills acquired from classroom environment. Their preference for the online session was only in as far as it complemented the traditional face-to-face learning. Learners in a study by Osgerby ( 2013 ) had positive perceptions of blended learning but preferred face-to-face with its step-by-stem instruction. Beard, Harper and Riley ( 2004 ) shows that some learners are successful while in a personal interaction with teachers and peers thus prefer face-to-face in the blend. Beard however dealt with a comparison between online and on-campus learning while our study combines both, singling out the face-to-face part of the blend. The advantage found by Beard is all the same relevant here because learners in blended learning express attitude to both online and face-to-face for an effective blend. Researchers indicate that teacher presence in face-to-face sessions lessens psychological distance between them and the learners and leads to greater learning. This is because there are verbal aspects like giving praise, soliciting for viewpoints, humor, etc and non-verbal expressions like eye contact, facial expressions, gestures, etc which make teachers to be closer to learners psychologically (Kelley & Gorham, 2009 ).

Learner outcomes

The outcomes under scrutiny in this study include performance, motivation, satisfaction and knowledge construction. Motivation is seen here as an outcome because, much as cognitive factors such as course grades are used in measuring learning outcomes, affective factors like intrinsic motivation may also be used to indicate outcomes of learning (Kuo, Walker, Belland, & Schroder, 2013 ). Research shows that high motivation among online learners leads to persistence in their courses (Menager-Beeley, 2004 ). Sankaran and Bui ( 2001 ) indicated that less motivated learners performed poorly in knowledge tests while those with high learning motivation demonstrate high performance in academics (Green, Nelson, Martin, & Marsh, 2006 ). Lim and Kim, ( 2003 ) indicated that learner interest as a motivation factor promotes learner involvement in learning and this could lead to learning effectiveness in blended learning.

Learner satisfaction was noted as a strong factor for effectiveness of blended and online courses (Wilging & Johnson, 2009) and dissatisfaction may result from learners’ incompetence in the use of the learning management system as an effective learning tool since, as Islam ( 2014 ) puts it, users may be dissatisfied with an information system due to ease of use. A lack of prompt feedback for learners from course instructors was found to cause dissatisfaction in an online graduate course. In addition, dissatisfaction resulted from technical difficulties as well as ambiguous course instruction Hara and Kling ( 2001 ). These factors, once addressed, can lead to learner satisfaction in e-learning and blended learning and eventual effectiveness. A study by Blocker and Tucker ( 2001 ) also showed that learners had difficulties with technology and inadequate group participation by peers leading to dissatisfaction within these design features. Student-teacher interactions are known to bring satisfaction within online courses. Study results by Swan ( 2001 ) indicated that student-teacher interaction strongly related with student satisfaction and high learner-learner interaction resulted in higher levels of course satisfaction. Descriptive results by Naaj, Nachouki, and Ankit ( 2012 ) showed that learners were satisfied with technology which was a video-conferencing component of blended learning with a mean of 3.7. The same study indicated student satisfaction with instructors at a mean of 3.8. Askar and Altun, ( 2008 ) found that learners were satisfied with face-to-face sessions of the blend with t-tests and ANOVA results indicating female scores as higher than for males in the satisfaction with face-to-face environment of the blended learning.

Studies comparing blended learning with traditional face-to-face have indicated that learners perform equally well in blended learning and their performance is unaffected by the delivery method (Kwak, Menezes, & Sherwood, 2013 ). In another study, learning experience and performance are known to improve when traditional course delivery is integrated with online learning (Stacey & Gerbic, 2007 ). Such improvement as noted may be an indicator of blended learning effectiveness. Our study however, delves into improved performance but seeks to establish the potential of blended learning effectiveness by considering grades obtained in a blended learning experiment. Score 50 and above is considered a pass in this study’s setting and learners scoring this and above will be considered to have passed. This will make our conclusions about the potential of blended learning effectiveness.

Regarding knowledge construction, it has been noted that effective learning occurs where learners are actively involved (Nurmela, Palonen, Lehtinen & Hakkarainen, 2003 , cited in Zhu, 2012 ) and this may be an indicator of learning environment effectiveness. Effective blended learning would require that learners are able to initiate, discover and accomplish the processes of knowledge construction as antecedents of blended learning effectiveness. A study by Rahman, Yasin and Jusoff ( 2011 ) indicated that learners were able to use some steps to construct meaning through an online discussion process through assignments given. In the process of giving and receiving among themselves, the authors noted that learners learned by writing what they understood. From our perspective, this can be considered to be accomplishment in the knowledge construction process. Their study further shows that learners construct meaning individually from assignments and this stage is referred to as pre-construction which for our study, is an aspect of discovery in the knowledge construction process.

Predictors of blended learning effectiveness

Researchers have dealt with success factors for online learning or those for traditional face-to-face learning but little is known about factors that predict blended learning effectiveness in view of learner characteristics and blended learning design features. This part of our study seeks to establish the learner characteristics/backgrounds and design features that predict blended learning effectiveness with regard to satisfaction, outcomes, motivation and knowledge construction. Song, Singleton, Hill, and Koh ( 2004 ) examined online learning effectiveness factors and found out that time management (a self-regulatory factor) was crucial for successful online learning. Eom, Wen, and Ashill ( 2006 ) using a survey found out that interaction, among other factors, was significant for learner satisfaction. Technical problems with regard to instructional design were a challenge to online learners thus not indicating effectiveness (Song et al., 2004 ), though the authors also indicated that descriptive statistics to a tune of 75% and time management (62%) impact on success of online learning. Arbaugh ( 2000 ) and Swan ( 2001 ) indicated that high levels of learner-instructor interaction are associated with high levels of user satisfaction and learning outcomes. A study by Naaj et al. ( 2012 ) indicated that technology and learner interactions, among other factors, influenced learner satisfaction in blended learning.

Objective and research questions of the current study

The objective of the current study is to investigate the effectiveness of blended learning in view of student satisfaction, knowledge construction, performance and intrinsic motivation and how they are related to student characteristics and blended learning design features in a blended learning environment.

Research questions

What are the student characteristics and blended learning design features for an effective blended learning environment?

Which factors (among the learner characteristics and blended learning design features) predict student satisfaction, learning outcomes, intrinsic motivation and knowledge construction?

Conceptual model of the present study

The reviewed literature clearly shows learner characteristics/background and blended learning design features play a part in blended learning effectiveness and some of them are significant predictors of effectiveness. The conceptual model for our study is depicted as follows (Fig.  1 ):

Conceptual model of the current study

Research design

This research applies a quantitative design where descriptive statistics are used for the student characteristics and design features data, t-tests for the age and gender variables to determine if they are significant in blended learning effectiveness and regression for predictors of blended learning effectiveness.

This study is based on an experiment in which learners participated during their study using face-to-face sessions and an on-line session of a blended learning design. A learning management system (Moodle) was used and learner characteristics/background and blended learning design features were measured in relation to learning effectiveness. It is therefore a planning evaluation research design as noted by Guskey ( 2000 ) since the outcomes are aimed at blended learning implementation at MMU. The plan under which the various variables were tested involved face-to-face study at the beginning of a 17 week semester which was followed by online teaching and learning in the second half of the semester. The last part of the semester was for another face-to-face to review work done during the online sessions and final semester examinations. A questionnaire with items on student characteristics, design features and learning outcomes was distributed among students from three schools and one directorate of postgraduate studies.

Participants

Cluster sampling was used to select a total of 238 learners to participate in this study. Out of the whole university population of students, three schools and one directorate were used. From these, one course unit was selected from each school and all the learners following the course unit were surveyed. In the school of Education ( n  = 70) and Business and Management Studies ( n  = 133), sophomore students were involved due to the fact that they have been introduced to ICT basics during their first year of study. Students of the third year were used from the department of technology in the School of Applied Sciences and Technology ( n  = 18) since most of the year two courses had a lot of practical aspects that could not be used for the online learning part. From the Postgraduate Directorate ( n  = 17), first and second year students were selected because learners attend a face-to-face session before they are given paper modules to study away from campus.

The study population comprised of 139 male students representing 58.4% and 99 females representing 41.6% with an average age of 24 years.

Instruments

The end of semester results were used to measure learner performance. The online self-regulated learning questionnaire (Barnard, Lan, To, Paton, & Lai, 2009 ) and the intrinsic motivation inventory (Deci & Ryan, 1982 ) were applied to measure the constructs on self regulation in the student characteristics and motivation in the learning outcome constructs. Other self-developed instruments were used for the other remaining variables of attitudes, computer competence, workload management, social and family support, satisfaction, knowledge construction, technology quality, interactions, learning management system tools and resources and face-to-face support.

Instrument reliability

Cronbach’s alpha was used to test reliability and the table below gives the results. All the scales and sub-scales had acceptable internal consistency reliabilities as shown in Table  1 below:

Data analysis

First, descriptive statistics was conducted. Shapiro-Wilk test was done to test normality of the data for it to qualify for parametric tests. The test results for normality of our data before the t- test resulted into significant levels (Male = .003, female = .000) thereby violating the normality assumption. We therefore used the skewness and curtosis results which were between −1.0 and +1.0 and assumed distribution to be sufficiently normal to qualify the data for a parametric test, (Pallant, 2010 ). An independent samples t -test was done to find out the differences in male and female performance to explain the gender characteristics in blended learning effectiveness. A one-way ANOVA between subjects was conducted to establish the differences in performance between age groups. Finally, multiple regression analysis was done between student variables and design elements with learning outcomes to determine the significant predictors for blended learning effectiveness.

Student characteristics, blended learning design features and learning outcomes ( RQ1 )

A t- test was carried out to establish the performance of male and female learners in the blended learning set up. This was aimed at finding out if male and female learners do perform equally well in blended learning given their different roles and responsibilities in society. It was found that male learners performed slightly better ( M  = 62.5) than their female counterparts ( M  = 61.1). An independent t -test revealed that the difference between the performances was not statistically significant ( t  = 1.569, df = 228, p  = 0.05, one tailed). The magnitude of the differences in the means is small with effect size ( d  = 0.18). A one way between subjects ANOVA was conducted on the performance of different age groups to establish the performance of learners of young and middle aged age groups (20–30, young & and 31–39, middle aged). This revealed a significant difference in performance (F(1,236 = 8.498, p < . 001).

Average percentages of the items making up the self regulated learning scale are used to report the findings about all the sub-scales in the learner characteristics/background scale. Results show that learner self-regulation was good enough at 72.3% in all the sub-scales of goal setting, environment structuring, task strategies, time management, help-seeking and self-evaluation among learners. The least in the scoring was task strategies at 67.7% and the highest was learner environment structuring at 76.3%. Learner attitude towards blended learning environment is at 76% in the sub-scales of learner autonomy, quality of instructional materials, course structure, course interface and interactions. The least scored here is attitude to course structure at 66% and their attitudes were high on learner autonomy and course interface both at 82%. Results on the learners’ computer competences are summarized in percentages in the table below (Table  2 ):

It can be seen that learners are skilled in word processing at 91%, email at 63.5%, spreadsheets at 68%, web browsers at 70.2% and html tools at 45.4%. They are therefore good enough in word processing and web browsing. Their computer confidence levels are reported at 75.3% and specifically feel very confident when it comes to working with a computer (85.7%). Levels of family and social support for learners during blended learning experiences are at 60.5 and 75% respectively. There is however a low score on learners being assisted by family members in situations of computer setbacks (33.2%) as 53.4% of the learners reported no assistance in this regard. A higher percentage (85.3%) is reported on learners getting support from family regarding provision of essentials for learning such as tuition. A big percentage of learners spend two hours on study while at home (35.3%) followed by one hour (28.2%) while only 9.7% spend more than three hours on study at home. Peers showed great care during the blended learning experience (81%) and their experiences were appreciated by the society (66%). Workload management by learners vis-à-vis studying is good at 60%. Learners reported that their workmates stand in for them at workplaces to enable them do their study in blended learning while 61% are encouraged by their bosses to go and improve their skills through further education and training. On the time spent on other activities not related to study, majority of the learners spend three hours (35%) while 19% spend 6 hours. Sixty percent of the learners have to answer to someone when they are not attending to other activities outside study compared to the 39.9% who do not and can therefore do study or those other activities.

The usability of the online system, tools and resources was below average as shown in the table below in percentages (Table  3 ):

However, learners became skilled at navigating around the learning management system (79%) and it was easy for them to locate course content, tools and resources needed such as course works, news, discussions and journal materials. They effectively used the communication tools (60%) and to work with peers by making posts (57%). They reported that online resources were well organized, user friendly and easy to access (71%) as well as well structured in a clear and understandable manner (72%). They therefore recommended the use of online resources for other course units in future (78%) because they were satisfied with them (64.3%). On the whole, the online resources were fine for the learners (67.2%) and useful as a learning resource (80%). The learners’ perceived usefulness/satisfaction with online system, tools, and resources was at 81% as the LMS tools helped them to communicate, work with peers and reflect on their learning (74%). They reported that using moodle helped them to learn new concepts, information and gaining skills (85.3%) as well as sharing what they knew or learned (76.4%). They enjoyed the course units (78%) and improved their skills with technology (89%).

Learner interactions were seen from three angles of cognitivism, collaborative learning and student-teacher interactions. Collaborative learning was average at 50% with low percentages in learners posting challenges to colleagues’ ideas online (34%) and posting ideas for colleagues to read online (37%). They however met oftentimes online (60%) and organized how they would work together in study during the face-to-face meetings (69%). The common form of communication medium frequently used by learners during the blended learning experience was by phone (34.5%) followed by whatsapp (21.8%), face book (21%), discussion board (11.8%) and email (10.9%). At the cognitive level, learners interacted with content at 72% by reading the posted content (81%), exchanging knowledge via the LMS (58.4%), participating in discussions on the forum (62%) and got course objectives and structure introduced during the face-to-face sessions (86%). Student-teacher interaction was reported at 71% through instructors individually working with them online (57.2%) and being well guided towards learning goals (81%). They did receive suggestions from instructors about resources to use in their learning (75.3%) and instructors provided learning input for them to come up with their own answers (71%).

The technology quality during the blended learning intervention was rated at 69% with availability of 72%, quality of the resources was at 68% with learners reporting that discussion boards gave right content necessary for study (71%) and the email exchanges containing relevant and much needed information (63.4%) as well as chats comprising of essential information to aid the learning (69%). Internet reliability was rated at 66% with a speed considered averagely good to facilitate online activities (63%). They however reported that there was intermittent breakdown during online study (67%) though they could complete their internet program during connection (63.4%). Learners eventually found it easy to download necessary materials for study in their blended learning experiences (71%).

Learner extent of use of the learning management system features was as shown in the table below in percentage (Table  4 ):

From the table, very rarely used features include the blog and wiki while very often used ones include the email, forum, chat and calendar.

The effectiveness of the LMS was rated at 79% by learners reporting that they found it useful (89%) and using it makes their learning activities much easier (75.2%). Moodle has helped learners to accomplish their learning tasks more quickly (74%) and that as a LMS, it is effective in teaching and learning (88%) with overall satisfaction levels at 68%. However, learners note challenges in the use of the LMS regarding its performance as having been problematic to them (57%) and only 8% of the learners reported navigation while 16% reported access as challenges.

Learner attitudes towards Face-to-face support were reported at 88% showing that the sessions were enjoyable experiences (89%) with high quality class discussions (86%) and therefore recommended that the sessions should continue in blended learning (89%). The frequency of the face-to-face sessions is shown in the table below as preferred by learners (Table  5 ).

Learners preferred face-to-face sessions after every month in the semester (33.6%) and at the beginning of the blended learning session only (27.7%).

Learners reported high intrinsic motivation levels with interest and enjoyment of tasks at 83.7%, perceived competence at 70.2%, effort/importance sub-scale at 80%, pressure/tension reported at 54%. The pressure percentage of 54% arises from learners feeling nervous (39.2%) and a lot of anxiety (53%) while 44% felt a lot of pressure during the blended learning experiences. Learners however reported the value/usefulness of blended learning at 91% with majority believing that studying online and face-to-face had value for them (93.3%) and were therefore willing to take part in blended learning (91.2%). They showed that it is beneficial for them (94%) and that it was an important way of studying (84.3%).

Learner satisfaction was reported at 81% especially with instructors (85%) high percentage reported on encouraging learner participation during the course of study 93%, course content (83%) with the highest being satisfaction with the good relationship between the objectives of the course units and the content (90%), technology (71%) with a high percentage on the fact that the platform was adequate for the online part of the learning (76%), interactions (75%) with participation in class at 79%, and face-to-face sessions (91%) with learner satisfaction high on face-to-face sessions being good enough for interaction and giving an overview of the courses when objectives were introduced at 92%.

Learners’ knowledge construction was reported at 78% with initiation and discovery scales scoring 84% with 88% specifically for discovering the learning points in the course units. The accomplishment scale in knowledge construction scored 71% and specifically the fact that learners were able to work together with group members to accomplish learning tasks throughout the study of the course units (79%). Learners developed reports from activities (67%), submitted solutions to discussion questions (68%) and did critique peer arguments (69%). Generally, learners performed well in blended learning in the final examination with an average pass of 62% and standard deviation of 7.5.

Significant predictors of blended learning effectiveness ( RQ 2)

A standard multiple regression analysis was done taking learner characteristics/background and design features as predictor variables and learning outcomes as criterion variables. The data was first tested to check if it met the linear regression test assumptions and results showed the correlations between the independent variables and each of the dependent variables (highest 0.62 and lowest 0.22) as not being too high, which indicated that multicollinearity was not a problem in our model. From the coefficients table, the VIF values ranged from 1.0 to 2.4, well below the cut off value of 10 and indicating no possibility of multicollinearity. The normal probability plot was seen to lie as a reasonably straight diagonal from bottom left to top right indicating normality of our data. Linearity was found suitable from the scatter plot of the standardized residuals and was rectangular in distribution. Outliers were no cause for concern in our data since we had only 1% of all cases falling outside 3.0 thus proving the data as a normally distributed sample. Our R -square values was at 0.525 meaning that the independent variables explained about 53% of the variance in overall satisfaction, motivation and knowledge construction of the learners. All the models explaining the three dependent variables of learner satisfaction, intrinsic motivation and knowledge construction were significant at the 0.000 probability level (Table  6 ).

From the table above, design features (technology quality and online tools and resources), and learner characteristics (attitudes to blended learning, self-regulation) were significant predictors of learner satisfaction in blended learning. This means that good technology with the features involved and the learner positive attitudes with capacity to do blended learning with self drive led to their satisfaction. The design features (technology quality, interactions) and learner characteristics (self regulation and social support), were found to be significant predictors of learner knowledge construction. This implies that learners’ capacity to go on their work by themselves supported by peers and high levels of interaction using the quality technology led them to construct their own ideas in blended learning. Design features (technology quality, online tools and resources as well as learner interactions) and learner characteristics (self regulation), significantly predicted the learners’ intrinsic motivation in blended learning suggesting that good technology, tools and high interaction levels with independence in learning led to learners being highly motivated. Finally, none of the independent variables considered under this study were predictors of learning outcomes (grade).

In this study we have investigated learning outcomes as dependent variables to establish if particular learner characteristics/backgrounds and design features are related to the outcomes for blended learning effectiveness and if they predict learning outcomes in blended learning. We took students from three schools out of five and one directorate of post-graduate studies at a Ugandan University. The study suggests that the characteristics and design features examined are good drivers towards an effective blended learning environment though a few of them predicted learning outcomes in blended learning.

Student characteristics/background, blended learning design features and learning outcomes

The learner characteristics, design features investigated are potentially important for an effective blended learning environment. Performance by gender shows a balance with no statistical differences between male and female. There are statistically significant differences ( p  < .005) in the performance between age groups with means of 62% for age group 20–30 and 67% for age group 31 –39. The indicators of self regulation exist as well as positive attitudes towards blended learning. Learners do well with word processing, e-mail, spreadsheets and web browsers but still lag below average in html tools. They show computer confidence at 75.3%; which gives prospects for an effective blended learning environment in regard to their computer competence and confidence. The levels of family and social support for learners stand at 61 and 75% respectively, indicating potential for blended learning to be effective. The learners’ balance between study and work is a drive factor towards blended learning effectiveness since their management of their workload vis a vis study time is at 60 and 61% of the learners are encouraged to go for study by their bosses. Learner satisfaction with the online system and its tools shows prospect for blended learning effectiveness but there are challenges in regard to locating course content and assignments, submitting their work and staying on a task during online study. Average collaborative, cognitive learning as well as learner-teacher interactions exist as important factors. Technology quality for effective blended learning is a potential for effectiveness though features like the blog and wiki are rarely used by learners. Face-to-face support is satisfactory and it should be conducted every month. There is high intrinsic motivation, satisfaction and knowledge construction as well as good performance in examinations ( M  = 62%, SD = 7.5); which indicates potentiality for blended learning effectiveness.

Significant predictors of blended learning effectiveness

Among the design features, technology quality, online tools and face-to-face support are predictors of learner satisfaction while learner characteristics of self regulation and attitudes to blended learning are predictors of satisfaction. Technology quality and interactions are the only design features predicting learner knowledge construction, while social support, among the learner backgrounds, is a predictor of knowledge construction. Self regulation as a learner characteristic is a predictor of knowledge construction. Self regulation is the only learner characteristic predicting intrinsic motivation in blended learning while technology quality, online tools and interactions are the design features predicting intrinsic motivation. However, all the independent variables are not significant predictors of learning performance in blended learning.

The high computer competences and confidence is an antecedent factor for blended learning effectiveness as noted by Hadad ( 2007 ) and this study finds learners confident and competent enough for the effectiveness of blended learning. A lack in computer skills causes failure in e-learning and blended learning as noted by Shraim and Khlaif ( 2010 ). From our study findings, this is no threat for blended learning our case as noted by our results. Contrary to Cohen et al. ( 2012 ) findings that learners’ family responsibilities and hours of employment can impede their process of learning, it is not the case here since they are drivers to the blended learning process. Time conflict, as compounded by family, employment status and management support (Packham et al., 2004 ) were noted as causes of learner failure and drop out of online courses. Our results show, on the contrary, that these factors are drivers for blended learning effectiveness because learners have a good balance between work and study and are supported by bosses to study. In agreement with Selim ( 2007 ), learner positive attitudes towards e-and blended learning environments are success factors. In line with Coldwell et al. ( 2008 ), no statistically significant differences exist between age groups. We however note that Coldwel, et al dealt with young, middle-aged and old above 45 years whereas we dealt with young and middle aged only.

Learner interactions at all levels are good enough and contrary to Astleitner, ( 2000 ) that their absence makes learners withdraw, they are a drive factor here. In line with Loukis (2007) the LMS quality, reliability and ease of use lead to learning efficiency as technology quality, online tools are predictors of learner satisfaction and intrinsic motivation. Face-to-face sessions should continue on a monthly basis as noted here and is in agreement with Marriot et al. ( 2004 ) who noted learner preference for it for facilitating social interaction and communication skills. High learner intrinsic motivation leads to persistence in online courses as noted by Menager-Beeley, ( 2004 ) and is high enough in our study. This implies a possibility of an effectiveness blended learning environment. The causes of learner dissatisfaction noted by Islam ( 2014 ) such as incompetence in the use of the LMS are contrary to our results in our study, while the one noted by Hara and Kling, ( 2001 ) as resulting from technical difficulties and ambiguous course instruction are no threat from our findings. Student-teacher interaction showed a relation with satisfaction according to Swan ( 2001 ) but is not a predictor in our study. Initiating knowledge construction by learners for blended learning effectiveness is exhibited in our findings and agrees with Rahman, Yasin and Jusof ( 2011 ). Our study has not agreed with Eom et al. ( 2006 ) who found learner interactions as predictors of learner satisfaction but agrees with Naaj et al. ( 2012 ) regarding technology as a predictor of learner satisfaction.

Conclusion and recommendations

An effective blended learning environment is necessary in undertaking innovative pedagogical approaches through the use of technology in teaching and learning. An examination of learner characteristics/background, design features and learning outcomes as factors for effectiveness can help to inform the design of effective learning environments that involve face-to-face sessions and online aspects. Most of the student characteristics and blended learning design features dealt with in this study are important factors for blended learning effectiveness. None of the independent variables were identified as significant predictors of student performance. These gaps are open for further investigation in order to understand if they can be significant predictors of blended learning effectiveness in a similar or different learning setting.

In planning to design and implement blended learning, we are mindful of the implications raised by this study which is a planning evaluation research for the design and eventual implementation of blended learning. Universities should be mindful of the interplay between the learner characteristics, design features and learning outcomes which are indicators of blended learning effectiveness. From this research, learners manifest high potential to take on blended learning more especially in regard to learner self-regulation exhibited. Blended learning is meant to increase learners’ levels of knowledge construction in order to create analytical skills in them. Learner ability to assess and critically evaluate knowledge sources is hereby established in our findings. This can go a long way in producing skilled learners who can be innovative graduates enough to satisfy employment demands through creativity and innovativeness. Technology being less of a shock to students gives potential for blended learning design. Universities and other institutions of learning should continue to emphasize blended learning approaches through installation of learning management systems along with strong internet to enable effective learning through technology especially in the developing world.

Abubakar, D. & Adetimirin. (2015). Influence of computer literacy on post-graduates’ use of e-resources in Nigerian University Libraries. Library Philosophy and Practice. From http://digitalcommons.unl.edu/libphilprac/ . Retrieved 18 Aug 2015.

Ahmad, N., & Al-Khanjari, Z. (2011). Effect of Moodle on learning: An Oman perception. International Journal of Digital Information and Wireless Communications (IJDIWC), 1 (4), 746–752.

Google Scholar  

Anderson, T. (2004). Theory and Practice of Online Learning . Canada: AU Press, Athabasca University.

Arbaugh, J. B. (2000). How classroom environment and student engagement affect learning in internet-basedMBAcourses. Business Communication Quarterly, 63 (4), 9–18.

Article   Google Scholar  

Askar, P. & Altun, A. (2008). Learner satisfaction on blended learning. E-Leader Krakow , 2008.

Astleitner, H. (2000) Dropout and distance education. A review of motivational and emotional strategies to reduce dropout in web-based distance education. In Neuwe Medien in Unterricht, Aus-und Weiterbildung Waxmann Munster, New York.

Barnard, L., Lan, W. Y., To, Y. M., Paton, V. O., & Lai, S. (2009). Measuring self regulation in online and blended learning environments’. Internet and Higher Education, 12 (1), 1–6.

Beard, L. A., Harper, C., & Riley, G. (2004). Online versus on-campus instruction: student attitudes & perceptions. TechTrends, 48 (6), 29–31.

Berenson, R., Boyles, G., & Weaver, A. (2008). Emotional intelligence as a predictor for success in online learning. International Review of Research in open & Distance Learning, 9 (2), 1–16.

Blocker, J. M., & Tucker, G. (2001). Using constructivist principles in designing and integrating online collaborative interactions. In F. Fuller & R. McBride (Eds.), Distance education. Proceedings of the Society for Information Technology & Teacher Education International Conference (pp. 32–36). ERIC Document Reproduction Service No. ED 457 822.

Cohen, K. E., Stage, F. K., Hammack, F. M., & Marcus, A. (2012). Persistence of master’s students in the United States: Developing and testing of a conceptual model . USA: PhD Dissertation, New York University.

Coldwell, J., Craig, A., Paterson, T., & Mustard, J. (2008). Online students: Relationships between participation, demographics and academic performance. The Electronic Journal of e-learning, 6 (1), 19–30.

Deci, E. L., & Ryan, R. M. (1982). Intrinsic Motivation Inventory. Available from selfdeterminationtheory.org/intrinsic-motivation-inventory/ . Accessed 2 Aug 2016.

Delone, W. H., & McLean, E. R. (2003). The Delone and McLean model of information systems success: A Ten-year update. Journal of Management Information Systems, 19 (4), 9–30.

Demirkol, M., & Kazu, I. Y. (2014). Effect of blended environment model on high school students’ academic achievement. The Turkish Online Journal of Educational Technology, 13 (1), 78–87.

Eom, S., Wen, H., & Ashill, N. (2006). The determinants of students’ perceived learning outcomes and satisfaction in university online education: an empirical investigation’. Decision Sciences Journal of Innovative Education, 4 (2), 215–235.

Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher education. Internet and Higher Education, 7 (2), 95–105.

Goyal, E., & Tambe, S. (2015). Effectiveness of Moodle-enabled blended learning in private Indian Business School teaching NICHE programs. The Online Journal of New Horizons in Education, 5 (2), 14–22.

Green, J., Nelson, G., Martin, A. J., & Marsh, H. (2006). The causal ordering of self-concept and academic motivation and its effect on academic achievement. International Education Journal, 7 (4), 534–546.

Guskey, T. R. (2000). Evaluating Professional Development . Thousands Oaks: Corwin Press.

Hadad, W. (2007). ICT-in-education toolkit reference handbook . InfoDev. from http://www.infodev.org/en/Publication.301.html . Retrieved 04 Aug 2015.

Hara, N. & Kling, R. (2001). Student distress in web-based distance education. Educause Quarterly. 3 (2001).

Heinich, R., Molenda, M., Russell, J. D., & Smaldino, S. E. (2001). Instructional Media and Technologies for Learning (7th ed.). Englewood Cliffs: Prentice-Hall.

Hofmann, J. (2014). Solutions to the top 10 challenges of blended learning. Top 10 challenges of blended learning. Available on cedma-europe.org .

Islam, A. K. M. N. (2014). Sources of satisfaction and dissatisfaction with a learning management system in post-adoption stage: A critical incident technique approach. Computers in Human Behaviour, 30 , 249–261.

Kelley, D. H. & Gorham, J. (2009) Effects of immediacy on recall of information. Communication Education, 37 (3), 198–207.

Kenney, J., & Newcombe, E. (2011). Adopting a blended learning approach: Challenges, encountered and lessons learned in an action research study. Journal of Asynchronous Learning Networks, 15 (1), 45–57.

Kintu, M. J., & Zhu, C. (2016). Student characteristics and learning outcomes in a blended learning environment intervention in a Ugandan University. Electronic Journal of e-Learning, 14 (3), 181–195.

Kuo, Y., Walker, A. E., Belland, B. R., & Schroder, L. E. E. (2013). A predictive study of student satisfaction in online education programs. International Review of Research in Open and Distributed Learning, 14 (1), 16–39.

Kwak, D. W., Menezes, F. M., & Sherwood, C. (2013). Assessing the impact of blended learning on student performance. Educational Technology & Society, 15 (1), 127–136.

Lim, D. H., & Kim, H. J. (2003). Motivation and learner characteristics affecting online learning and learning application. Journal of Educational Technology Systems, 31 (4), 423–439.

Lim, D. H., & Morris, M. L. (2009). Learner and instructional factors influencing learner outcomes within a blended learning environment. Educational Technology & Society, 12 (4), 282–293.

Lin, B., & Vassar, J. A. (2009). Determinants for success in online learning communities. International Journal of Web-based Communities, 5 (3), 340–350.

Loukis, E., Georgiou, S. & Pazalo, K. (2007). A value flow model for the evaluation of an e-learning service. ECIS, 2007 Proceedings, paper 175.

Lynch, R., & Dembo, M. (2004). The relationship between self regulation and online learning in a blended learning context. The International Review of Research in Open and Distributed Learning, 5 (2), 1–16.

Marriot, N., Marriot, P., & Selwyn. (2004). Accounting undergraduates’ changing use of ICT and their views on using the internet in higher education-A Research note. Accounting Education, 13 (4), 117–130.

Menager-Beeley, R. (2004). Web-based distance learning in a community college: The influence of task values on task choice, retention and commitment. (Doctoral dissertation, University of Southern California). Dissertation Abstracts International, 64 (9-A), 3191.

Naaj, M. A., Nachouki, M., & Ankit, A. (2012). Evaluating student satisfaction with blended learning in a gender-segregated environment. Journal of Information Technology Education: Research, 11 , 185–200.

Nurmela, K., Palonen, T., Lehtinen, E. & Hakkarainen, K. (2003). Developing tools for analysing CSCL process. In Wasson, B. Ludvigsen, S. & Hoppe, V. (eds), Designing for change in networked learning environments (pp 333–342). Dordrecht, The Netherlands, Kluwer.

Osgerby, J. (2013). Students’ perceptions of the introduction of a blended learning environment: An exploratory case study. Accounting Education, 22 (1), 85–99.

Oxford Group, (2013). Blended learning-current use, challenges and best practices. From http://www.kineo.com/m/0/blended-learning-report-202013.pdf . Accessed on 17 Mar 2016.

Packham, G., Jones, P., Miller, C., & Thomas, B. (2004). E-learning and retention key factors influencing student withdrawal. Education and Training, 46 (6–7), 335–342.

Pallant, J. (2010). SPSS Survival Mannual (4th ed.). Maidenhead: OUP McGraw-Hill.

Park, J.-H., & Choi, H. J. (2009). Factors influencing adult learners’ decision to drop out or persist in online learning. Educational Technology & Society, 12 (4), 207–217.

Picciano, A., & Seaman, J. (2007). K-12 online learning: A survey of U.S. school district administrators . New York, USA: Sloan-C.

Piccoli, G., Ahmad, R., & Ives, B. (2001). Web-based virtual learning environments: a research framework and a preliminary assessment of effectiveness in basic IT skill training. MIS Quarterly, 25 (4), 401–426.

Pituch, K. A., & Lee, Y. K. (2006). The influence of system characteristics on e-learning use. Computers & Education, 47 (2), 222–244.

Rahman, S. et al, (2011). Knowledge construction process in online learning. Middle East Journal of Scientific Research, 8 (2), 488–492.

Rovai, A. P. (2003). In search of higher persistence rates in distance education online programs. Computers & Education, 6 (1), 1–16.

Sankaran, S., & Bui, T. (2001). Impact of learning strategies and motivation on performance: A study in Web-based instruction. Journal of Instructional Psychology, 28 (3), 191–198.

Selim, H. M. (2007). Critical success factors for e-learning acceptance: Confirmatory factor models. Computers & Education, 49 (2), 396–413.

Shraim, K., & Khlaif, Z. N. (2010). An e-learning approach to secondary education in Palestine: opportunities and challenges. Information Technology for Development, 16 (3), 159–173.

Shrain, K. (2012). Moving towards e-learning paradigm: Readiness of higher education instructors in Palestine. International Journal on E-Learning, 11 (4), 441–463.

Song, L., Singleton, E. S., Hill, J. R., & Koh, M. H. (2004). Improving online learning: student perceptions of useful and challenging characteristics’. Internet and Higher Education, 7 (1), 59–70.

Stacey, E., & Gerbic, P. (2007). Teaching for blended learning: research perspectives from on-campus and distance students. Education and Information Technologies, 12 , 165–174.

Swan, K. (2001). Virtual interactivity: design factors affecting student satisfaction and perceived learning in asynchronous online courses. Distance Education, 22 (2), 306–331.

Article   MathSciNet   Google Scholar  

Thompson, E. (2004). Distance education drop-out: What can we do? In R. Pospisil & L. Willcoxson (Eds.), Learning Through Teaching (Proceedings of the 6th Annual Teaching Learning Forum, pp. 324–332). Perth, Australia: Murdoch University.

Tselios, N., Daskalakis, S., & Papadopoulou, M. (2011). Assessing the acceptance of a blended learning university course. Educational Technology & Society, 14 (2), 224–235.

Willging, P. A., & Johnson, S. D. (2009). Factors that influence students’ decision to drop-out of online courses. Journal of Asynchronous Learning Networks, 13 (3), 115–127.

Zhu, C. (2012). Student satisfaction, performance and knowledge construction in online collaborative learning. Educational Technology & Society, 15 (1), 127–137.

Zielinski, D. (2000). Can you keep learners online? Training, 37 (3), 64–75.

Download references

Authors’ contribution

MJK conceived the study idea, developed the conceptual framework, collected the data, analyzed it and wrote the article. CZ gave the technical advice concerning the write-up and advised on relevant corrections to be made before final submission. EK did the proof-reading of the article as well as language editing. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Author information

Authors and affiliations.

Mountains of the Moon University, P.O. Box 837, Fort Portal, Uganda

Mugenyi Justice Kintu & Edmond Kagambe

Vrije Universiteit Brussel, Pleinlaan 2, Brussels, 1050, Ixelles, Belgium

Mugenyi Justice Kintu & Chang Zhu

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Mugenyi Justice Kintu .

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Kintu, M.J., Zhu, C. & Kagambe, E. Blended learning effectiveness: the relationship between student characteristics, design features and outcomes. Int J Educ Technol High Educ 14 , 7 (2017). https://doi.org/10.1186/s41239-017-0043-4

Download citation

Received : 13 July 2016

Accepted : 23 November 2016

Published : 06 February 2017

DOI : https://doi.org/10.1186/s41239-017-0043-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Blended learning effectiveness
• Learner characteristics
• Design features
• Learning outcomes and significant predictors

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Front Psychol

The Role of Experiential Learning on Students’ Motivation and Classroom Engagement

Yangtao kong.

1 School of Education, Shaanxi Normal University, Xi’an, China

2 Faculty of Educational Science, Shaanxi Xueqian Normal University, Xi’an, China

Due to the birth of positive psychology in the process of education, classroom engagement has been flourished and got a remarkable role in the academic field. The other significant determining factor of success in education is motivation which is in line with classroom engagement. Moreover, based on the constructivist approach, experiential learning (EL) as a new method in education and a learner-centric pedagogy is at the center of attention, as a result of its contributions to improving the value of education which centers on developing abilities, and experiences. The current review makes an effort to consider the role of EL on students’ classroom engagement and motivation by inspecting its backgrounds and values. Subsequently, the efficacy of findings for academic experts in educational contexts is discussed.

Introduction

It is stated that a basic causative factor in the general achievement of learners studying in higher education is learners’ engagement ( Xerri et al., 2018 ; Derakhshan, 2021 ). It is extensively approved that learners who are actively participating in the learning progression and take interest in their academic education are more likely to achieve higher levels of learning ( Wang et al., 2021 ). Therefore, higher education institutions encourage learners to use their capabilities, as well as learning opportunities and facilities that enable them to be actively engaged ( Broido, 2014 ; Xie and Derakhshan, 2021 ). Moreover, students’ dissatisfaction, boredom, negative experiences, and dropping out of school are in part due to the low engagement in academic activities ( Derakhshan et al., 2021 ). It has been demonstrated that engagement is, directly and indirectly, related to intelligence, interest, motivation, and pleasure with learning outcomes within many academic fields ( Yin, 2018 ). Likewise, engagement is a construct that is shaped from the multifaceted relations of perceptions, feelings, and motivation which is corresponding to the progress of self-determination theory in the motivation realm ( Mercer and Dörnyei, 2020 ). Besides, the student’s motivation is a significant factor in cultivating learning and consequently increasing the value of higher education because the more the learners are motivated, the more likely they can be successful in their activities ( Derakhshan et al., 2020 ; Halif et al., 2020 ).

From a psychological point of view, motivating learners and engaging them in the classroom are closely related ( Han and Wang, 2021 ); nevertheless, motivation consists of factors that are psychological and difficult to observe, while engagement involves behaviors that can be observed by others that it is not simple to notice and estimate learners’ motivation ( Reeve, 2012 ). In other words, educators cannot concretely understand the fulfillment of their learners’ basic mental necessities and enthusiasm for learning ( Reeve, 2012 ). Nonetheless, Reeve asserted that in contrast to motivation, learners’ engagement by all accounts is a phenomenon that is distinctive and can nearly be noticed. Generally, educators can impartially consider whether or not a specific learner is engaged in the class exercises, such as problem solving.

As a reaction to the traditional teaching approach that is teacher-centric ( Che et al., 2021 ) and following the inclination to expanding interest in a more unique, participative learning atmosphere, educational organizations are orienting toward learning approaches that cultivate students’ involvement, interest, and dynamic participation. EL is a successful teaching method facilitating active learning through providing real-world experiences in which learners interact and critically evaluate course material and become involved with a topic being taught ( Boggu and Sundarsingh, 2019 ). Based on the teaching theory of Socrates, this model relies on research-based strategies which allow learners to apply their classroom knowledge to real-life situations to foster active learning, which consequently brings about a better retrieval ( Bradberry and De Maio, 2019 ). Indeed, engaging in daily activities, such as going to classes, completing schoolwork, and paying attention to the educator, is all indicators of classroom engagement ( Woods et al., 2019 ). Moreover, by participating in an EL class paired with relevant academic activities, learners improve their level of inherent motivation for learning ( Helle et al., 2007 ) and they have the opportunity to choose multiple paths to solve problems throughout the learning process by having choices and being autonomous ( Svinicki and McKeachie, 2014 ). EL is regarded as learning by action whereby information is built by the student during the renovation of changes ( Afida et al., 2012 ). Within EL, people become remarkably more liable for their learning which regulates a stronger connection between the learning involvement, practices, and reality ( Salas et al., 2009 ) that are key roles in learning motivation.

To make sure that the learners gain the required knowledge and get the factual training, it is equally important to give them time to develop their ability to use their knowledge and apply those skills in real-world situations to resolve problems that are relevant to their careers ( Huang and Jiang, 2020 ). So, it seems that they would like more hands-on training and skills development, but awkwardly, in reality, they generally just receive theoretical and academic education ( Green et al., 2017 ). In addition, as in today’s modern world, where shrewd and high-performing people are required, motivation and engagement should be prioritized in educational institutions as they are required features in the learning setting while they are often overlooked in classrooms ( Afzali and Izadpanah, 2021 ). Even though studies on motivation, engagement, and EL have been conducted so far; however, based on the researcher’s knowledge, just some have currently carried out systematic reviews about the issue and these studies have not been all taken together to date; therefore, concerning this gap, the current mini-review tries to take their roles into account in education.

Classroom Engagement and Motivation

As a three-dimensional construct, classroom engagement can be classified into three types: physical, emotional, and psychological ( Rangvid, 2018 ). However, it is not always easy to tell whether a learner is engaged because observable indicators are not always accurate. Even those who display signs of curiosity or interest in a subject or who seem engaged may not acquire knowledge about it. Others may also be learning despite not displaying any signs of physical engagement ( Winsett et al., 2016 ).

As an important component of success and wellbeing, motivation encourages self-awareness in individuals by inspiring them ( Gelona, 2011 ). Besides, it is a power that manages, encourages, and promotes goal-oriented behavior, which is not only crucial to the process of learning but also essential to educational achievement ( Kosgeroglu et al., 2009 ). It appears that classroom motivation is influenced by at least five factors: the learner, the educator, the course content, the teaching method, and the learning environment ( D’Souza and Maheshwari, 2010 ).

Experiential Learning

EL, developed by Kolb in 1984, is a paradigm for resolving the contradiction between how information is gathered and how it is used. It is focused on learning through experience and evaluating learners in line with their previous experiences ( Sternberg and Zhang, 2014 ). The paradigm highlights the importance of learners’ participation in all learning processes and tackles the idea of how experience contributes to learning ( Zhai et al., 2017 ). EL is a method of teaching that allows learners to learn while “Do, Reflect, and Think and Apply” ( Butler et al., 2019 , p. 12). Students take part in a tangible experience (Do), replicate that experience and other evidence (Reflect), cultivate theories in line with experiences and information (Think), and articulate an assumption or elucidate a problem (Apply). It is a strong instrument for bringing about positive modifications in academic education which allow learners to apply what they have learned in school to real-world problems ( Guo et al., 2016 ). This way of learning entails giving learners more authority and responsibility, as well as involving them directly in their learning process within the learning atmosphere. Furthermore, it encourages learners to be flexible learners, incorporate all possible ways of learning into full-cycle learning, and bring about effective skills and meta-learning abilities ( Kolb and Kolb, 2017 ).

Implications and Future Directions

This review focused on the importance of EL and its contributions to classroom engagement and motivation. Since experiential education tends to engage a wider range of participants who can have an impact on the organization, employees, educators, leaders, and future colleagues, it is critical to maintain its positive, welcoming atmosphere. The importance of EL lies in its ability to facilitate connections between undergraduate education and professional experience ( Earnest et al., 2016 ), so improving the connection between the university and the world of work ( Friedman and Goldbaum, 2016 ).

The positive effect of EL has actual implications for teachers who are thinking of implementing this method in their classes; indeed, they can guarantee their learners’ success by providing them with the knowledge required in performing the task as following the experiential theory, knowledge is built through converting practice into understanding. Based on the literature review, the conventional role of the teacher shifts from knowledge provider to a mediator of experience through well-known systematic processes. Likewise, teachers should encourage learners by providing information, suggestion, and also relevant experiences for learning to build a learning milieu where they can be engaged in positive but challenging learning activities that facilitate learners’ interaction with learning materials ( Anwar and Qadir, 2017 ) and illustrates their interest and motivation toward being a member of the learning progression. By learners’ dynamic participation in experiential activities, the teacher can trigger their ability to retain knowledge that leads to their intrinsic motivation and interest in the course material ( Zelechoski et al., 2017 ).

The present review is significant for the learners as it allows them to model the appropriate behavior and procedures in real-life situations by putting the theory into practice. Indeed, this method helps learners think further than memorization to evaluate and use knowledge, reflecting on how learning can be best applied to real-world situations ( Zelechoski et al., 2017 ). In the context of EL, students often find activities challenging and time-consuming which necessitates working in a group, performing work outside of the classroom, learning and integrating subject content to make decisions, adapt procedures, compare, and contrast various resources of information to detect a difficulty at one hand and implement that information on the other hand to form a product that aims to solve the issue. Participation, interaction, and application are fundamental characteristics of EL. During the process, it is possible to be in touch with the environment and to be exposed to extremely flexible processes. In this way, education takes place on all dimensions which cover not only the cognitive but also the affective and behavioral dimensions to encompass the whole person. Learners enthusiastically participate in mental, emotional, and social interactions during the learning procedure within EL ( Voukelatou, 2019 ). In addition, learners are encouraged to think logically, find solutions, and take appropriate action in relevant situations. This kind of instruction not only provides opportunities for discussion and clarification of concepts and knowledge, but also provides feedback, review, and transfer of knowledge and abilities to new contexts.

Moreover, for materials developers and syllabus designers to truly start addressing the learners’ motivation and engagement, they could embrace some interesting and challenging activities because when they can find themselves successful in comprehending the issue and being able to apply their information to solve it; they are not only more interested to engage in the mental processes required for obtaining knowledge but also more motivated and eager to learn. More studies can be conducted to investigate the effect of EL within different fields of the study courses with a control group design to carry out between-group comparisons. Besides, qualitative research is recommended to scrutinize the kinds of EL activities which make a more considerable effect on the EFL learners’ motivation and success and even their achievement.

Author Contributions

The author confirms being the sole contributor of this work and has approved it for publication.

This study was funded by the Projects of National Philosophy Social Science Fund, PRC (17CRK008), and the Projects of Philosophy and Social Science Fund of Shaanxi Province, PRC (2018Q11).

Conflict of Interest

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

• Afida A., Rosadah A. M., Aini H., Mohd Marzuki M. (2012). Creativity enhancement through experiential learning . Adv. Nat. Appl. Sci. 6 , 94–99. [ Google Scholar ]
• Afzali Z., Izadpanah S. (2021). The effect of the flipped classroom model on Iranian English foreign language learners: engagement and motivation in English language grammar . Cogent Educ. 8 :1870801. doi: 10.1080/2331186X.2020.1870801 [ CrossRef ] [ Google Scholar ]
• Anwar K., Qadir G. H. (2017). A study of the relationship between work engagement and job satisfaction in private companies in Kurdistan . Int. J. Adv. Eng. Manage. Sci. 3 , 1102–1110. doi: 10.24001/ijaems.3.12.3 [ CrossRef ] [ Google Scholar ]
• Boggu A. T., Sundarsingh J. (2019). An experiential learning approach to fostering learner autonomy among Omani students . J. Lang. Teach. Res. 10 , 204–214. doi: 10.17507/jltr.1001.23 [ CrossRef ] [ Google Scholar ]
• Bradberry L. A., De Maio J. (2019). Learning by doing: The long-term impact of experiential learning programs on student success . J. Political Sci. Educ. 15 , 94–111. doi: 10.1080/15512169.2018.1485571 [ CrossRef ] [ Google Scholar ]
• Broido E. M. (2014). Book review: one size does not fit all: traditional and innovative models of student affairs practice . J. Stud. Aff. Afr. 2 , 93–96. doi: 10.14426/jsaa.v2i1.52 [ CrossRef ] [ Google Scholar ]
• Butler M. G., Church K. S., Spencer A. W. (2019). Do, reflect, think, apply: experiential education in accounting . J. Acc. Educ. 48 , 12–21. doi: 10.1016/j.jaccedu.2019.05.001 [ CrossRef ] [ Google Scholar ]
• Che F. N., Strang K. D., Vajjhala N. R. (2021). Using experiential learning to improve student attitude and learning quality in software engineering education . Int. J. Innovative Teach. Learn. Higher Educ. 2 , 1–22. doi: 10.4018/IJITLHE.20210101.oa2 [ CrossRef ] [ Google Scholar ]
• D’Souza K. A., Maheshwari S. K. (2010). Factors influencing student performance in the introductory management science course . Acad. Educ. Leadersh. J. 14 , 99–119. [ Google Scholar ]
• Derakhshan A. (2021). The predictability of Turkman students’ academic engagement through Persian language teachers’ nonverbal immediacy and credibility . J. Teach. Persian Speakers Other Lang. 10 , 3–26. [ Google Scholar ]
• Derakhshan A., Coombe C., Arabmofrad A., Taghizadeh M. (2020). Investigating the effects of English language teachers’ professional identity and autonomy in their success . Issue Lang. Teach. 9 , 1–28. doi: 10.22054/ilt.2020.52263.496 [ CrossRef ] [ Google Scholar ]
• Derakhshan A., Kruk M., Mehdizadeh M., Pawlak M. (2021). Boredom in online classes in the Iranian EFL context: sources and solutions . System 101 :102556. doi: 10.1016/j.system.2021.102556 [ CrossRef ] [ Google Scholar ]
• Earnest D., Rosenbusch K., Wallace-Williams D., Keim A. (2016). Study abroad in psychology: increasing cultural competencies through experiential learning . Teach. Psychol. 43 , 75–79. doi: 10.1177/0098628315620889 [ CrossRef ] [ Google Scholar ]
• Friedman F., Goldbaum C. (2016). Experiential learning: developing insights about working with older adults . Clin. Soc. Work. J. 44 , 186–197. doi: 10.1007/s10615-016-0583-4 [ CrossRef ] [ Google Scholar ]
• Gelona J. (2011). Does thinking about motivation boost motivation levels . Coaching Psychol. 7 , 42–48. [ Google Scholar ]
• Green R. A., Conlon E. G., Morrissey S. M. (2017). Task values and self-efficacy beliefs of undergraduate psychology students . Aust. J. Psychol. 69 , 112–120. doi: 10.1111/ajpy.12125 [ CrossRef ] [ Google Scholar ]
• Guo F., Yao M., Wang C., Yang W., Zong X. (2016). The effects of service learning on student problem solving: the mediating role of classroom engagement . Teach. Psychol. 43 , 16–21. doi: 10.1177/0098628315620064 [ CrossRef ] [ Google Scholar ]
• Halif M. M., Hassan N., Sumardi N. A., Omar A. S., Ali S., Aziz R. A., et al.. (2020). Moderating effects of student motivation on the relationship between learning styles and student engagement . Asian J. Univ. Educ. 16 , 94–103. [ Google Scholar ]
• Han Y., Wang Y. (2021). Investigating the correlation among Chinese EFL teachers’ self-efficacy, work engagement, and reflection . Front. Psychol. 12 :763234. doi: 10.3389/fpsyg.2021.763234, PMID: [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Helle L., Tynjälä P., Olkinuora E., Lonka K. (2007). ‘Ain’t nothing like the real thing.’ Motivation and study processes on a work-based project course in information systems design . Br. J. Educ. Psychol. 77 , 397–411. doi: 10.1348/000709906X105986, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Huang R., Jiang L. (2020). Authentic assessment in Chinese secondary English classrooms: teachers’ perception and practice . Educ. Stud. , 1–14. doi: 10.1080/03055698.2020.1719387 [ CrossRef ] [ Google Scholar ]
• Kolb A. Y., Kolb D. A. (2017). Experiential learning theory as a guide for experiential educators in higher education . Exp. Learn. Teach. Higher Educ. 1 , 7–44. [ Google Scholar ]
• Kosgeroglu N., Acat M. B., Ayranci U., Ozabaci N., Erkal S. (2009). An investigation on nursing, midwifery and health care students’ learning motivation in Turkey . Nurse Educ. Pract. 9 , 331–339. doi: 10.1016/j.nepr.2008.07.003, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Mercer S., Dörnyei Z. (2020). Engaging Language Learners in Contemporary Classrooms. Cambridge: Cambridge University Press. [ Google Scholar ]
• Rangvid B. S. (2018). Student engagement in inclusive classrooms . Educ. Econ. 26 , 266–284. doi: 10.1080/09645292.2018.1426733 [ CrossRef ] [ Google Scholar ]
• Reeve J. (2012). “ A self-determination theory perspective on student engagement ,” in Handbook of Research on Student Engagement. eds. Christenson S. L., Reschly A. L., Wylie C. (New York, NY: Springer; ), 149–172. [ Google Scholar ]
• Salas E., Wildman J. L., Piccolo R. F. (2009). Using simulation-based training to enhance management education . Acad. Manage. Learn. Educ. 8 , 559–573. doi: 10.5465/amle.8.4.zqr559 [ CrossRef ] [ Google Scholar ]
• Sternberg R. J., Zhang L. F. (2014). Perspectives on Thinking, Learning and Cognitive Styles. Mahwah, NJ: Lawrence Erlbaum Associates. [ Google Scholar ]
• Svinicki M. D., McKeachie W. J. (2014). McKeachie’s Teaching Tips: Strategies, Research, and Theory for College and University Teachers. 14th Edn . Belmont, CA: Wadsworth Cengage Learning. [ Google Scholar ]
• Voukelatou G. (2019). The contribution of experiential learning to the development of cognitive and social skills in secondary education: A case study . Educ. Sci. 9 , 127–138. doi: 10.3390/educsci9020127 [ CrossRef ] [ Google Scholar ]
• Wang Y., Derakhshan A., Zhang L. J. (2021). Researching and practicing positive psychology in second/foreign language learning and teaching: The past, current status and future directions . Front. Psychol. 12:731721. 12 :731721. doi: 10.3389/fpsyg.2021.731721 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Winsett C., Foster C., Dearing J., Burch G. (2016). The impact of group experiential learning on student engagement . Acad. Bus. Res. J. 3 , 7–17. [ Google Scholar ]
• Woods A. D., Price T., Crosby G. (2019). The impact of the student-athlete’s engagement strategies on learning, development, and retention: A literary study . Coll. Stud. J. 53 , 285–292. [ Google Scholar ]
• Xerri M. J., Radford K., Shacklock K. (2018). Student engagement in academic activities: A social support perspective . High. Educ. 75 , 589–605. doi: 10.1007/s10734-017-0162-9 [ CrossRef ] [ Google Scholar ]
• Xie F., Derakhshan A. (2021). A conceptual review of positive teacher interpersonal communication behaviors in the instructional context . Front. Psychol. 12 , 1–10. doi: 10.3389/fpsyg.2021.708490, PMID: [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Yin H. (2018). What motivates Chinese undergraduates to engage in learning? Insights from a psychological approach to student engagement research . High. Educ. 76 , 827–847. doi: 10.1007/s10734-018-0239-0 [ CrossRef ] [ Google Scholar ]
• Zelechoski A. D., Riggs Romaine C. L., Wolbransky M. (2017). Teaching psychology and law . Teach. Psychol. 44 , 222–231. doi: 10.1177/0098628317711316 [ CrossRef ] [ Google Scholar ]
• Zhai X., Gu J., Liu H., Liang J.-C., Tsai C.-C. (2017). An experiential learning perspective on students’ satisfaction model in a flipped classroom context . Educ. Technol. Soc. 20 , 198–210. [ Google Scholar ]

Change Password

Your password must have 8 characters or more and contain 3 of the following:.

• a lower case character, 
• an upper case character, 
• a special character 

Password Changed Successfully

Your password has been changed

• Sign in / Register

Request Username

Can't sign in? Forgot your username?

Enter your email address below and we will send you your username

If the address matches an existing account you will receive an email with instructions to retrieve your username

Engaging Undergraduate Students in Course-based Research Improved Student Learning of Course Material

• Nicole T. Appel
• Ammar Tanveer
• Sara Brownell
• Joseph N. Blattman

School of Life Sciences, Arizona State University, Tempe, AZ 85281

Search for more papers by this author

*Address correspondence to: Joseph N. Blattman ( E-mail Address: [email protected] ).

Course-based undergraduate research experiences (CUREs) offer students opportunities to engage in critical thinking and problem solving. However, quantitating the impact of incorporating research into undergraduate courses on student learning and performance has been difficult since most CUREs lack a comparable traditional course as a control. To determine how course-based research impacts student performance, we compared summative assessments of the traditional format for our upper division immunology course (2013–2016), in which students studied known immune development and responses, to the CURE format (2017–2019), in which students studied the effects of genetic mutations on immune development and responses. Because the overall class structure remained unaltered, we were able to quantitate the impact of incorporating research on student performance. Students in the CURE format class performed significantly better on quizzes, exams, and reports. There were no significant differences in academic levels, degree programs, or grade point averages, suggesting improved performance was due to increased engagement of students in research.

INTRODUCTION

Research experiences benefit undergraduate students by offering opportunities to engage in critical thinking and problem solving beyond the textbook and known experimental outcomes ( Kardash, 2000 ; Russell et al. , 2007 ; D’Arcy et al. , 2019 ). AAAS Vision and Change: A Call to Action suggested incorporating research into undergraduate education for students to appreciate the setbacks and unexpected outcomes of scientific research and to apply analytical skills and critical thinking to understanding their results (Bauerle et al. , 2011). Skills developed during undergraduate research experiences (UREs), such as teamwork, critical thinking, and oral and written communication, help prepare students for the workforce independent of whether they stay in a Science, Technology, Engineering, and Mathematics (STEM) field ( McClure-Brenchley et al. , 2020 ). Participating in undergraduate research is also associated with increased retention and likelihood of pursuing scientific research as a career ( Mastronardi et al. , 2021 ). Course-based undergraduate research experiences (CURE) provide students the opportunities to obtain scientific process skills while also having a greater outreach compared with one-on-one undergraduate research experiences ( Bangera and Brownell, 2014 ; Burmeister et al. , 2021 ). CURE classes are defined as integrating scientific practices, discovery, collaboration, and iteration with broadly relevant work. All five criteria must be met for a class to be a CURE, although every CURE may cover each criterion in varying degrees ( Auchincloss et al. , 2014 ).

Various surveys for undergraduate research experiences are available for measuring psychological and knowledge-based gains of participating in a CURE class. Using these tools, research has shown several benefits to transitioning to a CURE class. Biology Intensive Orientation Summer (BIOS) is a CURE that originated in China to help undergraduate students gain research skills and graduate students gain mentor skills. The CURE gave students the confidence and skills to pursue mentor based UREs, which was a graduation requirement ( Fendos et al. , 2022 ). Another study found that, after participating in a CURE for a year, students perceived gains in scientific literacy, data collection, presenting results via oral and written communication, and maintaining a lab notebook ( Peteroy-Kelly et al. , 2017 ). Other CUREs, such as the biochemistry authentic scientific inquiry laboratory (BASIL) CURE, measured student reported gains in lab skills, aka anticipated learning outcomes or course-based undergraduate research abilities. Students were asked about their knowledge, experience, and confidence in the seven anticipated learning outcomes in pretests and posttests and reported gains in all seven areas ( Irby et al. , 2020 ). Measuring gains in content knowledge is rarer, but a study by Wolkow et al. followed up with students 1 and 3 years after taking an introductory biology class to which students were randomly assigned to either a CURE or traditional lab class ( Wolkow et al. , 2014 ; Wolkow et al. , 2019 ). One year after the course, students in the CURE lab reported greater psychological gains, such as enjoying the class and considering a research career, and performed better on an assessment used to measure gains on topics covered in the CURE lab. The knowledge gains for general introductory biology were comparable between groups ( Wolkow et al. , 2014 ). By senior year, perceived gains were no longer different between those who were in the CURE and traditional lab classes as freshmen, and the knowledge gains for general introductory biology were comparable, too. However, the targeted knowledge gains of what was covered in the lab classes remained significantly higher in the CURE group ( Wolkow et al. , 2019 ).

In many CUREs, students develop their own questions and experiments to fulfill the five criteria of integrating scientific practices, discovery, collaboration, and iteration with broadly relevant work. Although students report positive outcomes when asked to compare CUREs with previous traditional labs they have taken, obtaining empirical, measurable benefits for students to engage in undergraduate research is difficult when questions and experiments vary by semester or even by lab group ( Linn et al. , 2015 ). In collaboration with Dr. Brownell, we agreed members from her lab could interview our students to determine whether any differences in cognitive and emotional ownership existed between the two class formats and, if so, whether that impacted student perceptions on collaboration, iteration, or discovery/relevance. The interviews were performed in 2016 (traditional), 2017 (CURE), and 2018 (CURE). Based on that collaboration, we learned that changing our Experimental Immunology class to a CURE format did not significantly impact collaboration or iteration, but the CURE format students perceived their data were novel and relevant outside of class. CURE format students also expressed increased cognitive and emotional ownership compared with traditional format students ( Cooper et al. , 2019 ). To ease the transition from a traditional format to the CURE format, the only change made between the formats was that the CURE students researched how a genetic change impacted the immune response alongside the control experiments by comparing the known immune responses of wild-type (WT) mice to previously uncharacterized genetically modified mice. The experiments and assessments were unchanged between class formats. We realized retrospectively that we were in a unique position to empirically measure whether and how incorporating research impacted students and therefore fulfill the gap in knowledge detailed by Linn et al. (2015) . We knew our students had increased cognitive and emotional ownership when research was incorporated into the course ( Cooper et al. , 2019 ), and ownership has been linked to improved student performance ( Martínez et al. , 2019 ). Therefore, the first question we asked was whether incorporating research into the course resulted in improved overall performance? This question was examined using three main variables: overall course performance; sets of quizzes, reports, and exams; and individual assessment items. While we could not quantitate the amount of literature read or scientific skills acquired, we were able to compare the scores of the direct summative assessments intended to measure student learning. In this lab course, we had direct summative assessments in the form of lab reports, quizzes, and two exams. During the semester, we also assessed participation and lab notebooks to encourage students to come prepared to perform experiments and to understand the material beforehand, but we did not use participation or notebook grades to measure how well students learned the course material since those assessments were tools to ensure students came to class knowing the procedures and actively participated. Therefore, we compared the total results from direct summative assessments (lab report, quiz, and exam grades) to determine whether incorporating research into a lab class resulted in any impact on student learning. The quizzes and exams remained identical between the two class formats, which provided another control variable when comparing class formats. Because the Teaching Assistants (TAs) grading the assessments did not know scores would be compared by the professor after the transition to a CURE format, we argue the assessments were graded without bias for CURE or traditional formats. Because students were not told beforehand whether the class was traditional or CURE format, the “volunteer effect” also did not impact our findings ( Brownell et al. , 2013 ). In a second experimental question, we asked whether additional factors influenced course performance or were distinct between traditional versus CURE formats, including grade point average (GPA), academic major, academic year, grading trends, and racial/ethnic or gender diversity.

MATERIALS AND METHODS

This study was conducted with an approved Institutional Review Board protocol (#4249) from Arizona State University.

Traditional Class Format

The class was divided into five sections each consisting of two or three laboratory exercises. The five sections were anatomy and cells of the immune system (labs 2–3), innate immunity (labs 4–6), adaptive immune system development (labs 7–8), acute adaptive immune response (labs 10–12), and immune memory and protection (labs 13–15). All experiments and protocols followed the lab manual “The Immune System: An Experimental Approach” ( Blattman et al. , 2016 ). The purpose of each lab could be copied from the lab manual. In other words, the traditional lab class format was prescriptive or “cookbook.” While students wrote individual hypotheses, the findings were not novel and no outside literature was needed to support the hypothesis; all necessary information to form a hypothesis was in the lab manual. All lab experiments were performed using immune cells from recently killed WT Bl6 mice (IACUC 19-1684T). The class structure was to start with a quiz, review the quiz, answer students’ questions regarding immunology and the day’s lab exercises, and then let the students perform the experiments and, when applicable, gather data the same day. Notebooks were signed at the end of class. Students were encouraged to have everything written in their notebooks and review class material before class started.

CURE Class Format

To transition to a CURE format, students studied mice with a genetic mutation that had not been studied in immunology thereby generating novel data. By collaborating with other laboratories within Arizona State University, students studied what effect knocking out Mohawk (2017, Alan Rawls), having a Raf1L613V mutation (2018, Jason Newborn), or knocking out Z-DNA-binding protein 1 (2019, Bertram Jacobs) had on the immune response. The five areas of immunology studied, the lab manual, and protocols remained unchanged between traditional and CURE formats. The purpose of each CURE lab shifted to understanding how the immune response of a genetically modified mouse differed from the WT mouse. CURE students were required to read outside literature before class started to generate a novel hypothesis. They were instructed to hypothesize how the immune response would differ (better, worse, or no change) between mice and provide their own reasoning as to why. Other than encouraging students to find publications on pubmed, instructors did not help students generate hypotheses. Before experiments started, students discussed their hypotheses in small groups before sharing their different hypotheses with the class. Instructors encouraged students to share their different hypotheses by asking, “Did anyone think the immune response would be better in the knock out mouse? Why? Who thought there’d be no change? Why?” and finally saying, “All these hypotheses are valid. We don’t know the answer yet because the experiment has never been done before.” The class structure was to start with the quiz, review the quiz, answer immunology questions, discuss hypotheses and reasoning, answer questions related to lab exercise, let the students perform the experiments and, when applicable, gather data the same day. Notebooks were signed at the end of class. Because there was no time to generate a hypothesis during class, students were required to read the course material and apply it to the genetic mutation before coming to class.

At the beginning of every lab, students took a quiz on the immunology on which the lab was based and the experiment itself. The first and last quizzes were omitted because the first quiz was used to show students how the class would flow throughout the semester and therefore did not count toward the final grade and the last quiz was a practical to determine student ability to analyze flow cytometry data.

Before class, both teaching methods required students to read the lab material, write the purpose, question, hypothesis, and procedures for the day’s lab in their notebooks, and take a quiz at the beginning of class. Students used the same book with the same optional practice questions and took the same quizzes. Although both class formats required students to come to class prepared, the CURE teaching method enforced that requirement because incorporating real scientific research into the class required CURE students to develop a novel hypothesis on how altering the gene of interest would impact the immune response. Due to time constraints, all reading for generating their novel hypothesis needed to happen before the class started. Immediately after reviewing the quiz, CURE students discussed their hypotheses in their groups for 2 minutes prior to sharing with the class via random call. To receive a notebook signature for the day’s lab, CURE students needed to have citations for their hypotheses. Students were expected to have hypotheses with citations before the start of class beginning with the second lab quiz.

Students wrote lab reports after the first four class sections. The fifth lab report was not included in this analysis since students taking the class 2013–2015 were not told to write a fifth report, and students in 2016–2019 could write the fifth report to replace the lowest report grade. Therefore, the significant difference between report grades was calculated based on reports 1 through 4 without replacing any scores since report 5 was omitted. Regardless of class format, students followed the same report rubric. Each report was worth 50 points, and the points were Introduction-7, Methods-5, Results-12, Discussion-15, References-5, Grammar-2, Legends/captions-2, and Formatting-2. In the introduction, students were expected to provide relevant information, purpose of the experiments, questions answered, and hypotheses. The CURE students not only provided the relevant immunology background for the report but also read additional literature for relevant background information regarding the gene of interest. This background reading (which took place before class and therefore before the quiz) then needed to provide a clear link to the hypothesis. Students were told the hypothesis should answer whether they expected the immune response would be greater, the same, or less than the WT mouse and why. The methods remained unchanged other than the CURE students had one additional sample to run due to also analyzing the immune response from the genetically modified mouse. For results, students in both class formats analyzed immune organ cell counts, flow cytometry data, ELISA results, and cytotoxicity data for their reports. However, students taking the CURE format had additional samples and needed to compare WT results with the genetically altered mouse. While the analysis itself was similar given the rubric (figures, description/summary, how data were obtained, and identifying controls in the experiments), CURE students analyzed two sets of data, learned how to prevent bias between samples, and then compared/contrasted the data in the discussion. In the discussion, both class formats read outside literature and discussed the impact of the data. Both formats analyzed WT data and determined whether the data obtained fit within expected values. For the CURE students, the WT data served as a control that then told them whether they correctly performed the experiment. Therefore, if the WT values fit the expected norm, then the data from the genetically altered mouse, which used the same methods and reagents, could be believed. Students then read further literature to try to understand the reasoning behind the results from the genetically altered mouse and then showed how their research regarding the gene of interest had impact outside of class. Both class formats discussed the impact of the established immunology and why the immunology was important to study (Supplemental Table S1).

The class had two exams: the open book take-home midterm was given as a hard copy before spring break and due when classes resumed and the closed book in-class final.

The students followed the same rubric for lab reports (Supplemental Table S1) and had the same quizzes and exams. Quiz and exam questions consisted of multiple choice, fill in the blank, drawing, short answer, identify cells or organs, and math. Short answer and drawing questions could have resulted in variance in TA grading. However, any variance was mitigated by reviewing all quiz answers in class and exam answers when requested. The professor, who did not change between formats, was present for classes and answered questions regarding which short answers were or were not acceptable and whether partial credit would be granted. Drawings were also reviewed in class using either a whiteboard or TV screens depending on class size.

If the increase in grades in the CURE format were due to students obtaining copies of previous quizzes, then we would expect quiz grades to have started rising during the 4 years the traditional format was taught. The midterm exam was always an open book take home exam given before students left for spring break. The final exam was a closed book, in class exam for both class formats and had the same questions and available number of points.

For both formats, TAs encouraged study practices for the final exam. In the traditional class, students were allowed a notecard during the final. In the CURE format, students had an in-class quiz-like review session 2 days before the final. While the TAs provided different study aids, students still studied on their own. In other words, the students in the traditional class could have still quizzed themselves while students in the CURE class were observed taking notes during the review session.

Regarding lab reports, different TAs graded the lab reports depending on the year. However, the same rubric was followed for grading, and the available number of points for each report remained consistent within the class format.

Finally, the quality of education remained consistent across the different class formats. The same professor was responsible for the class even though the TA teaching the class changed. In both formats, the class TAs were recognized for quality teaching. The traditional format was taught by a TA who was student-nominated and awarded ASU’s Teacher of the Year. The CURE format was taught by a TA who was self-nominated and awarded GPSA’s Teaching Excellence Award.

Student Demographics

GPA, degree program, and academic level were all analyzed in the results section as described below. The class did not have any prerequisites to enroll and was not required by any degree program at Arizona State University. In other words, the likelihood of students enrolling in Experimental Immunology remained consistent between class formats. A lecture class (MIC 420: Basic Immunology) was offered all the years that Experimental Immunology was taught. However, the lecture class was not required, and both the traditional and CURE class formats had a mixture of students who had and had not taken the lecture. Students did not elect to enroll in a CURE or traditional class and were not told prior to enrollment that the class format had changed to a CURE. Other demographics, including prior research experience, were assessed previously and not found to be significantly different between class formats ( Cooper et al. , 2019 ).

Statistical Analysis

Scores from quizzes, reports, and exams were pulled from 2013 to 2019 and analyzed for any differences via GraphPad Prism unpaired t tests. Correction from multiple t tests was done using false discovery rate determined by two-stage step-up (Benjamini, Krieger, and Yekutieli). GPA was also analyzed via unpaired t tests to determine significance.

The Shannon–Wiener diversity index, Chi-squared, and Fisher t test were used to determine level of diversity for degree program, academic level, race, and gender. The Shannon–Wiener diversity index is a way to measure diversity within a population ( Shannon, 1948 ). A t test was used to compare results from the Shannon–Wiener diversity index ( Hutcheson, 1970 ). Further analysis for degree program, academic level, and race used Chi-squared. Significant differences in gender were determined using the Fisher t test.

GraphPad Prism’s multiple regression analysis was used to determine the impact of the predictor variables GPA, class format (0-Traditional, 1-Cure), and academic level on the outcome variable (overall points earned). Further analysis studied the predictor variables on points earned on quizzes, reports, and exams separately.

Scores from quizzes 2013–2016 were analyzed via one-way ANOVA in GraphPad Prism.

Students in CURE Class Averaged ∼5% Higher than Students Taught via Traditional Method

To determine whether students learned more course material in the CURE format, we compared overall course grades and found that students in the CURE class performed better overall with a class average of 80% compared with students in the traditional format course who averaged 75% ( p < 0.0001) ( Figure 1A ). The aggregate semester grade was calculated from student quizzes (79% vs. 71%, p < 0.0001) ( Figure 1B ), lab reports (84% vs. 78%, p < 0.0001) ( Figure 1C ), and exams (82% vs. 77%, p < 0.01) ( Figure 1D ). On all three assessments, CURE format students had significantly improved performance, which suggests incorporating research into the immunology laboratory class resulted in improved understanding and application of the course material.

FIGURE 1. Changing class format to a CURE improved student performance. (A) Students enrolled in the CURE class format demonstrated improved mastery of course material compared with those in the traditional class format based on improved scores in quizzes, lab reports, and exams. (B) Quizzes were given at the beginning of class before the instructor/TA reviewed the material and experimental setup. Based on quiz scores, CURE students demonstrated increased understanding of and preparedness for class. (C) Lab reports assessed scientific writing and ability to analyze and interpret data. Students enrolled in the CURE format performed better overall on reports. (D) Students engaged in research scored higher on exams indicating improved mastery of course material. CURE students n = 139, traditional students n = 119; **** p < 0.0001, ** p < 0.01; unpaired t test was used to test statistical significance with false discovery rate determined by two-stage step-up (Benjamini, Krieger, and Yekutieli).

Changing Class Format to a CURE Improved Majority of Quiz Scores

We used quizzes to test student preparedness for class and understanding of important background information each laboratory period. CURE students achieved significantly higher scores on seven out of 13 quizzes ( Figure 2 ). Early in the semester, the CURE teaching method resulted in students performing significantly better on the second quiz (88% vs. 80%, p < 0.01). The CURE teaching method resulted in continued improved performance when viral infection was mimicked in quiz 5 by studying the innate immune response to poly(I:C) (61% vs. 51%, p < 0.01) and when lymphocyte development was studied in quiz 7 (88% vs. 78%, p < 0.0001) and quiz 8 (88% vs. 81%. p < 0.01). Later in the semester, lymphocyte response to virus was studied, specifically how T cells respond to viral infections. The difference in quiz scores between teaching methods then often exceeded 15% such as in quizzes 10 (76% vs. 58%, p < 0.0001), 11 (82% vs. 61%, p < 0.0001), and 14 (83% vs. 66%, p < 0.0001). Scores for quizzes 12 and 13 were not significantly different between teaching methods (70% vs. 62% and 80% vs. 75%, respectively). When the focus was on learning a new technique instead of forming a new hypothesis, such as quiz 6 (79% vs. 80%) and quiz 9 (77% vs. 75%), no significant difference in scores was noticed. Scores for quizzes 3 and 4 scores were also not significantly different between teaching methods (87% vs. 82% and 67% vs. 71%, respectively). Lab 3 studied cells of the immune system and reviewed fundamentals for flow cytometry, which the class used to analyze data. Lab 4 studied oxidative burst, which occurs when leukocytes encounter a pathogen. Overall, seven of the 13 quizzes were significantly improved for CURE format versus traditional format. Of the six quizzes that were not significantly improved, two involved learning a technique instead of generating a hypothesis before obtaining novel results.

FIGURE 2. Incorporating research into the course resulted in improved quiz scores. Of the quizzes analyzed, students engaged in research earned higher scores in seven of the 13 quizzes. Two of the four quizzes in which there was no significant difference did not have novel data in the lab classes for those quizzes. Unfilled bars represent the traditional format, and filled bars represent the CURE format. CURE students n = 139, traditional students n = 119; **** p < 0.0001, ** p < 0.01, * p < 0.05, ns = not significant; unpaired t test was used to test statistical significance with false discovery rate determined by two-stage step-up (Benjamini, Krieger, and Yekutieli).

Incorporating Scientific Research Resulted in Improved Performance on Reports

While quizzes demonstrated student preparedness for class, we used laboratory reports to assess student analytical skills for interpreting data, as well as critical thinking about how their work applied to current research outside the class. The rubric for grading reports was unchanged between class formats. We found significantly improved scores for all four analyzed laboratory reports from CURE format students compared with scores from traditional format students. As with quizzes, incorporating research into the class benefitted students from the beginning. The CURE format resulted in students earning 6% higher on the first report (74% vs. 68%, p < 0.01). Students appeared to incorporate feedback from the first report regardless of class format given the second report was one letter grade higher for both sets of students. However, the benefit of incorporating research early resulted in the CURE class still scoring 8% higher (87% vs. 79%, p < 0.0001). The third report (89% vs. 83%, p < 0.0001) and fourth report (87% vs. 84%, p < 0.05) also demonstrated improved scientific writing when research was incorporated ( Figure 3 ).

FIGURE 3. Students demonstrated better scientific writing when they produced and analyzed novel data. Lab reports assessed analytical skills and data interpretation and required students to look at contemporary literature to understand how their work was applicable outside class. CURE students were told from the beginning of the semester their work was novel. The same rubric was used for both class formats in which over half the grade came from the results and discussion sections. CURE students scored higher on all analyzed reports. Unfilled bars represent the traditional format, and filled bars represent the CURE format. CURE students n = 139, traditional students n = 119; **** p < 0.0001, ** p < 0.01, * p < 0.05; unpaired t test was used to test statistical significance with false discovery rate determined by two-stage step-up (Benjamini, Krieger, and Yekutieli).

Midterm, but not Final, Exam Scores Improved in CURE Format

Two exams were given to assess student mastery of the course material. A midterm exam assessed student understanding of labs 1–9, and a final exam covered material from labs 10 to 15. The questions and format were the same for exams for the traditional and CURE format courses. Again, students in the CURE format class performed significantly better on the midterm exam compared with students from the traditional format course (88% vs. 83%, p < 0.0001). However, student performance on the final exam did not differ significantly between traditional and CURE format courses (75% vs. 73%) ( Figure 4 ).

FIGURE 4. CURE students scored higher on the midterm but not the final. The exams tested student understanding of the course material; no questions were modified to incorporate research material. All students were provided with the same take-home, open-book midterm to be completed in the same timeframe. Although course-based research was not incorporated into the exam itself, CURE students scored higher on the midterm. The final exam was administered in class after all experiments were completed. CURE students and traditional students performed equally on the final. Unfilled bars represent the traditional format, and filled bars represent the CURE format. CURE students n = 139, traditional students n = 119; **** p < 0.0001, ns = not significant; unpaired t test was used to test statistical significance with false discovery rate determined by two-stage step-up (Benjamini, Krieger, and Yekutieli).

Student Demographics Remained Consistent Between Formats

The improved performance on quizzes, exams, and lab reports in the CURE format course compared with the traditional format course, despite no other differences in format or assessments, suggests incorporating research into a laboratory course increases student mastery. However, other factors including student demographics could also result in this change. To determine whether the student population changed, data on students’ overall academic GPAs, degree programs, and academic levels were analyzed. No significant difference was observed across GPA ( p = 0.07) ( Figure 5C ), degree program ( p = 0.6) ( Figure 5A ), or academic level ( p = 0.4) ( Figure 5B ). Therefore, the students who took the traditional class format were equally capable of mastering the course material as students who took the CURE class format. The improved performance observed in the CURE format was due to incorporating research into the teaching method.

FIGURE 5. Student demographics were unchanged between traditional and CURE formats. (A) Students enrolled in either the traditional or CURE format participated in similar degree programs. CURE students n = 139, traditional students n = 119; Shannon Diversity test followed by an unpaired t test was used to test statistical significance in differences between formats. (B) Both the traditional and CURE formats consisted mostly of seniors. No significant difference was observed in student academic level between the different formats. CURE students n = 139, traditional students n = 119; Shannon Diversity test followed by an unpaired t test was used to test statistical significance in differences between formats. (C) No significant difference was found in student GPAs between the class formats. CURE students n = 139, traditional students n = 119; ns = not significant; an unpaired t test was used to test statistical significance .

Multiple Linear Regression Analysis Indicates the Teaching Intervention Improved Student Scores

Multiple linear regression (MLR) controls for other variables that impact student performance and is therefore a reliable method for determining whether a teaching intervention, such as incorporating research, impacted student performance and learning or whether the change in performance was due to student-intrinsic factors ( Theobald and Freeman, 2014 ). In setting up the MLR analysis, we chose student GPA, class format, and academic level as the predictor, or control, variables. The outcome variable was the total number of points earned in the class. GPA represented overall academic performance. Academic level helped measure preparedness of previous coursework since more senior students would likely have taken more life science classes to prepare them for an upper division immunology course. Class format represented the teaching intervention, which was incorporating research. MLR analysis showed how well incorporating research impacted student performance ( p = 0.0004) in the class when the predictor variables were controlled ( Table 1 ). GPA also served as a good indicator of well a student would do in the class ( p < 0.0001), but academic level had no impact on how well students performed in class. Individual analyses were performed for quizzes, reports, and exams with similar findings (Supplemental Tables S2–S4).

Improved Scores were Due to Changing the Teaching Format without Changing Assessments

If the increase in grades in the CURE format were due to students obtaining copies of previous assessments, especially quizzes, then we hypothesized that we would see significant increases in quiz grades prior to changing class formats. We analyzed quiz averages across the 4 years the traditional lab was taught, 2013–2016. While there was an improvement between 2013 and 2014 (Supplemental Figure S1), no further improvement was noted across the 4 years. 2013 was the first year this course was taught. Nonetheless, we reanalyzed the overall scores for quizzes, reports, and exams to determine whether 2013 falsely lowered the scores from the traditional class format. All three assessment areas remained significant between class formats when the scores from the first ever class were omitted (Supplemental Figure S2).

Assessing Differences in TA Grading Showed No Significance Difference between Formats

The traditional format had four graders over the course of 4 years (two TAs and two assistant TAs). The CURE format had two graders over the course of 3 years (two TAs). The class professor, who was consistent across formats, also graded occasionally. Although unlikely that the clear divide in grades across class formats was due to grading variances since there were seven total graders for the class, we sought to determine whether differences in scores were due to grading. We therefore analyzed the coefficient of variance (%CV) within samples and compared the two class formats. Any %CV due to student-intrinsic factors would be similar between teaching methods because student demographics were comparable between class formats. If %CV were significantly different between formats, then other factors, including differences in TA grading, would likely be responsible. The %CV for quizzes, reports, and exams were not significant between class formats ( p = 0.0671, 0.3162, and 0.8858, respectively) ( Figure 6 ), which suggests that the grading rigor was comparable between class formats.

FIGURE 6. Grading practices were comparable between traditional and CURE formats. (A) The 13 quizzes were analyzed for %CV to determine intravariability in grading in both traditional and CURE formats. The %CVs were then analyzed via unpaired t test. No significant difference in intravariability was found between class formats. (B) The four reports were analyzed for %CV to determine intravariability in grading in both traditional and CURE formats. The %CVs were then analyzed via unpaired t test. No significant difference in intravariability was found between class formats. (C) The two exams were analyzed for %CV to determine intravariability in grading in both traditional and CURE formats. The %CVs were then analyzed via unpaired t test. No significant difference in intravariability was found between class formats.

Experimental Immunology Taught a Diverse Student Population

CUREs are known to include more students in research and have a broader outreach than the traditional one-on-one mentoring method ( Bangera and Brownell, 2014 ; Burmeister et al. , 2021 ). Both the traditional and CURE class formats rated high on the Shannon–Wiener diversity index with richness scores of 7 and 11, respectively. There was no significant difference between class diversity as calculated via Shannon–Wiener diversity index ( p = 0.2) or Chi-squared test ( p = 0.3255) ( Figure 7A ). Both the traditional and CURE class formats had over 50% female students, and the ratio of female to male students was not significantly different between class formats as calculated via Shannon–Wiener diversity index ( p = 0.2) or Fisher’s exact test ( p = 0.1298) ( Figure 7B ). Overall, the Experimental Immunology class serves a diverse group of students. By changing the class format to incorporate research, we included the diverse student population we serve in critical thinking and problem solving.

FIGURE 7. Traditional and CURE formats served equally diverse student populations. (A) Students enrolled in the course came from diverse racial backgrounds, several of which are underrepresented in science. CURE students n = 139, traditional students n = 119; Shannon diversity test followed by an unpaired t test was used to test statistical significance in differences between formats. Chi-squared test was also tested. No significant difference was observed between class formats. (B) More women than men enrolled in Experimental Immunology. CURE students n = 139, traditional students n = 119; Shannon diversity test followed by an unpaired t test was used to test statistical significance in differences between formats. Fisher’s exact test was also used. No significant difference was observed between class formats.

While incorporating research into existing laboratory courses benefits students by encouraging critical thinking, problem solving, and reading current literature to show how their work is novel and applicable outside class, quantitating the impact of integrating research on student mastery of the course material has been difficult ( Linn et al. , 2015 ). We changed the format of an upper division immunology lab course into a CURE class by having students study the immune response of previously uncharacterized genetically altered mice compared with the known response of WT mice. The class structure, such as the experiments performed, lab manual used, and assessments, remained unchanged between formats thereby allowing us to compare the effect incorporating research has on student performance in the class. We realized retroactively that we were therefore in a unique position to determine whether incorporating research improved student learning of the original course material as evidenced by improved scores.

Overall, we found incorporating research into the class resulted in students performing significantly better in all assessment areas. The overall difference in student performance was not surprising since every assessment showed that incorporating research improved student performance. The difference in quiz scores could be due to the nature of the hypotheses required for both classes. Because students studied known outcomes in the traditional class format, the lab manual often provided enough information for students to know what to expect and why. However, the lab manual did not detail any genetic mutations. While CURE students would have read the same immunology background from the lab manual, they had the additional responsibility to apply what they learned to whether a genetic mutation would impact the immune system and, if so, how. Encouraging students to apply their knowledge before taking any assessment likely resulted in improved learning of the course material and therefore higher quiz scores ( Freeman et al. , 2014 ).

Incorporating research improved students’ scientific writing as evidenced by improved lab report scores. Writing about real scientific results in the results and discussion sections, which are responsible for over half the lab report grade, likely made scientific writing more approachable. For example, the discussion section required students to compare their results with outside literature and explain why their results did or did not agree with current literature. The CURE teaching method required students to start reading outside literature before writing the report. Therefore, CURE students had an advantage regarding which literature sources to cite for the discussion because they already read multiple sources to formulate a hypothesis. The increased ownership and engagement in the class ( Cooper et al. , 2019 ) may have also resulted in increased scores as they had to write why their novel results were important outside of class ( Conley and French, 2014 ; Cannata et al. , 2019 ; Martínez et al. , 2019 ). The traditional teaching format encouraged outside reading before writing the report but did not require it. The results were not novel for the traditional teaching method, and therefore the impact of what was studied focused less on their results and more on how the experiments studied are still used in current research.

The significant difference in exam scores resulted from the CURE teaching method improving student scores by 5% on the midterm exam. Because the midterm exam was an open book take home exam completed over spring break, students in both class formats had equal access to the lab manual and previous quizzes to do equally well on the exam. Therefore, the difference in scores was likely due to student motivation to devote the time to do well on the exam ( Dweck, 1986 ) possibly due to increased project ownership ( Cannata et al. , 2017 ). The midterm also correlated with the increased peak in quiz performance suggesting that students in the CURE format exhibited higher levels of engagement immediately after spring break.

The difference in scores began to decline when quizzes and assignments for the class overlapped with projects necessary for students to graduate, such as capstone and honors thesis projects. If students experienced equal levels of burnout or had multiple assignments due around the time the final was taken, then student engagement in the class would be comparable between class formats and therefore explain why the scores on the final exam were not significantly different.

Active learning encourages students to take ownership of their education by actively participating in what they learn. By changing the lab format to a CURE class, students received guidance on how to look up and interpret journal articles. Once empowered in ways to educate themselves and look up information beyond what was provided in the course materials, the improved scores suggest students truly engaged in the class and took ownership of their projects and their education. Teaching the students scientific processes, such as graphing, data analysis, experimental design, scientific writing, and science communication, before students enrolled in introductory science classes improved content learning when students later enrolled in introductory biology classes despite minimal differences in student GPA/SAT scores ( Dirks and Cunningham, 2006 ). CUREs teach scientific processes alongside class material ( Auchincloss et al. , 2014 ). Our work supports that CUREs are an effective way to improve undergraduate education by engaging more students in scientific research. We showed transitioning to a CURE format resulted in a similar improvement in scores compared with teaching scientific processes separately. This further shows the CURE format enhanced student learning of course material rather than distracted from it, which is a concern raised when educators express reasons for not integrating more active learning in their courses ( Kim et al. , 2018 ; Shadle et al. , 2017 ; Ul-Huda et al. , 2018 ).

One limitation of this study is that different TAs were present and responsibilities, such as grading, shifted as TAs shifted. To mitigate variation, answers were reviewed in class whenever possible (always for quizzes, upon request for exams). Regarding quizzes, since all those answers were approved by the professor who was present for both class formats, CURE students performed better than traditional students in seven of the 13 quizzes, or seven of the 11 quizzes that involved applying their knowledge to a genetic mutation. A subset of assessments was unavailable to regrade to verify the impact of adding research because assessments were handed back to students. Nonetheless, most questions did not have multiple correct answers. The only questions that could have different points awarded based on TA leniency were drawings and short answers. To understand whether there were any significant differences, we compared %CV between formats. We showed through multiple analyses (MLR, Chi-squared, diversity index, and previous work from Cooper et al. (2019) ) that student demographics were not responsible for the difference in scores. Any %CV related to student ability would remain consistent between formats. We reasoned that a change in %CV would therefore be due to other variables, such as TA grading leniency. Our data showed that %CV was not different between class formats. Therefore, we do not believe having different graders significantly impacted the scores because no variance was noted within the class formats themselves.

We also analyzed whether academic dishonesty in the form of obtaining quizzes from prior years could have resulted in improved scores. While the first year the class was taught did have lower scores compared with subsequent semesters, no further improvement occurred. The improvement from 2013 to 2014 likely resulted from having an experienced professor and experienced TAs.

Another limitation of this study is we had no information on differences in family support and/or responsibilities (parents, spouse, children), socioeconomics such as whether the students were working to support themselves or were supported by family, and other personal factors that could affect student performance in the class ( Mushtaq and Khan, 2012 ). Nonetheless, previous studies showed prior test scores, course knowledge, and experience had the highest correlations to student performance when compared with other factors such as learning/teaching styles, gender, and family stress ( Van Lanen et al. , 2000 ; Clark and Latshaw, 2012 ; Mushtaq and Khan, 2012 ). Because no significant difference was noted in GPA, degree program, or academic level between the two formats, the improved scores likely resulted from the teaching method and not the students. This was further supported via MLR analysis in which class format was found to contribute to student performance when all other variables, including GPA, were controlled. GPA was accessed when most of the students had graduated Arizona State University, which means the GPA analyzed was likely the students’ final GPAs or close to their final GPAs. We recognize that our students had both visible and invisible diversities that were not disclosed in interviews or through demographic information. Therefore, measuring to what extent transitioning the class format to a CURE class impacted each demographic is outside the scope of this study. Nonetheless, we did note that students in the CURE class performed better overall. We hope this information encourages educators to include active learning, particularly course-based research, in their classes and universities to offer rewards and incentives for educators to update courses as needed to improve student engagement and thereby improve student mastery of course material.

Overall, we showed incorporating research into an upper division lab improved student learning and mastery of course material. Changing the class to a CURE format resulted in students experiencing increased project ownership, which was likely associated with increased engagement in the course and ownership of their education which then translated to improved scores on assessments. We were able to show this because the overall class structure and assessments remained unaltered between the traditional and CURE class formats; the only change was students studied how a previously uncharacterized gene impacted immune system development, response, and memory. Over the course of 3 years, 139 students from diverse backgrounds, some of which are underrepresented in science, participated in scientific research through this class, which supports that CUREs can engage large numbers of diverse students in science ( Auchincloss et al. , 2014 ).

ACKNOWLEDGMENTS

We would like to thank all the labs that collaborated with Experimental Immunology to provide genetically modified mice. While the labs were already breeding mice for their own research, we are grateful for the work they put in to breed additional mice for this class. Special thanks to Cherie Alissa Lynch (2017), Michael Holter (2018), and Karen Kibler (2019) for helping make the CURE class possible. Student fees for Arizona State University’s MIC 421 Experimental Immunology class 435 were used to fund the class experiments.

• Auchincloss, L. C., Laursen, S. L., Branchaw, J. L., Eagan, K., Graham, M., Hanauer, D. I., … & Dolan, E. L. ( 2014 ). Assessment of course-based undergraduate research experiences: A meeting report . CBE—Life Sciences Education , 13 (1), 29–40.  https://doi.org/10.1187/cbe.14-01-0004 Link ,  Google Scholar
• Bangera, G., & Brownell, S. E. ( 2014 ). Course-based undergraduate research experiences can make scientific research more inclusive . CBE—Life Sciences Education , 13 (4), 602–606.  https://doi.org/10.1187/cbe.14-06-0099 Link ,  Google Scholar
• Bauerle, C. M. , American Association for the Advancement of Science , National Science Foundation (U.S.) , Division of Undergraduate Education , & Directorate for Biological Sciences . ( 2011 ) Vision and change in undergraduate biology education: A call to action: Final report of a national conference held from July 15–17, Washington, DC . Google Scholar
• Blattman, J. N., McAfee, M. S., & Schoettle, L. ( 2016 ). The Immune System: An Experimental Approach (Preliminary) . Cognella, Inc. Google Scholar
• Brownell, S. E., Kloser, M. J., Fukami, T., & Shavelson, R. J. ( 2013 ). Context matters: Volunteer bias, small sample size, and the value of comparison groups in the assessment of research-based undergraduate introductory biology lab courses . Journal of Microbiology & Biology Education , 14 (2), 176–182.  https://doi.org/10.1128/jmbe.v14i2.609 Medline ,  Google Scholar
• Burmeister, A. R., Dickinson, K., & Graham, M. J. ( 2021 ). Bridging trade-offs between traditional and course-based undergraduate research experiences by building student communication skills, identity, and interest . Journal of Microbiology & Biology Education , 22 (2).  https://doi.org/10.1128/jmbe.00156-21 Google Scholar
• Cannata, M., Redding, C., & Nguyen, T. ( 2019 ). Building student ownership and responsibility: Examining student outcomes from a research-practice partnership . Journal of Research on Education Effectiveness , 12 (3), 333–362. Google Scholar
• Cannata, M. A., Smith, T. M., & Taylor Haynes, K. ( 2017 ). Integrating academic press and support by increasing student ownership and responsibility . AERA Open , 3 (3), 233285841771318.  https://doi.org/10.1177/2332858417713181 Google Scholar
• Clark, S., & Latshaw, C. ( 2012 ). “Peeling the Onion” called student performance: An investigation into the factors affecting student performance in an introductory accounting class . Review of Business; New York , 33 (1), 19–27. Google Scholar
• Conley, D. T., & French, E. M. ( 2014 ). Student ownership of learning as a key component of college readiness . American Behavioral Scientist , 58 (8), 1018–1034.  https://doi.org/10.1177/0002764213515232 Google Scholar
• Cooper, K. M., Blattman, J. N., Hendrix, T., & Brownell, S. E. ( 2019 ). The impact of broadly relevant novel discoveries on student project ownership in a traditional lab course turned CURE . CBE—Life Sciences Education , 18 (4), ar57.  https://doi.org/10.1187/cbe.19-06-0113 Link ,  Google Scholar
• D’Arcy, C. E., Martinez, A., Khan, A. M., & Olimpo, J. T. ( 2019 ). Cognitive and non-cognitive outcomes associated with student engagement in a novel brain chemoarchitecture mapping course-based undergraduate research experience . Journal of Undergraduate Neuroscience Education: JUNE: A Publication of FUN, Faculty for Undergraduate Neuroscience , 18 (1), A15–A43 Medline ,  Google Scholar
• Dirks, C., & Cunningham, M. ( 2006 ). Enhancing diversity in science: Is teaching science process skills the answer? CBE—Life Sciences Education , 5 (3), 218–226.  https://doi.org/10.1187/cbe.05-10-0121 Link ,  Google Scholar
• Dweck, C. S. ( 1986 ). Motivational processes affecting learning . American Psychologist , 41 (10), 1040–1048.  https://doi.org/10.1037/0003-066X.41.10.1040 Google Scholar
• Fendos, J., Cai, L., Yang, X., Ren, G., Li, L., Yan, Z., Lu, B., Pi, Y., Ma, J., Guo, B., Wu, X., Lu, P., Zhang, R., & Yang, J. ( 2022 ). A course-based undergraduate research experience improves outcomes in mentored research . CBE—Life Sciences Education , 21 (3), ar49.  https://doi.org/10.1187/cbe.21-03-0065 Medline ,  Google Scholar
• Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. ( 2014 ). Active learning increases student performance in science, engineering, and mathematics . Proceedings of the National Academy of Sciences , 111 (23), 8410–8415. Medline ,  Google Scholar
• Hutcheson, K. ( 1970 ). A test for comparing diversities based on the shannon formula . Journal of Theoretical Biology , 29 (1), 151–154.  https://doi.org/10.1016/0022-5193(70)90124-4 Google Scholar
• Irby, S. M., Pelaez, N. J., & Anderson, T. R. ( 2020 ). Student perceptions of their gains in course-based undergraduate research abilities identified as the anticipated learning outcomes for a biochemistry CURE . Journal of Chemical Education , 97 (1), 56–65.  https://doi.org/10.1021/acs.jchemed.9b00440 Google Scholar
• Kardash, C. M. ( 2000 ). Evaluation of undergraduate research experience: Perceptions of undergraduate interns and their faculty mentors . Journal of Educational Psychology , 92 (1), 191–201.  https://doi.org/10.1037/0022-0663.92.1.191 Google Scholar
• Kim, A. M., Speed, C. J., & Macaulay, J. O. ( 2018 ). Barriers and strategies: Implementing active learning in biomedical science lectures . Biochemistry and Molecular Biology Education , 47 (1), 29–40.  https://doi.org/10.1002/bmb.21190 Google Scholar
• Linn, M. C., Palmer, E., Baranger, A., Gerard, E., & Stone, E. ( 2015 ). Undergraduate research experiences: Impacts and opportunities . Science , 347 (6222), 1261757. https://doi.org/10.1126/science.1261757 Medline ,  Google Scholar
• Martínez, I. M., Youssef-Morgan, C. M., Chambel, M. J., & Marques-Pinto, A. ( 2019 ). Antecedents of academic performance of university students: Academic engagement and psychological capital resources . Educational Psychology , 39 (8), 1047–1067.  https://doi.org/10.1080/01443410.2019.1623382 Google Scholar
• Mastronardi, Borrego, M., Choe, N., & Hartman, R. ( 2021 ). The impact of undergraduate research experiences on participants’ career decisions . Journal of STEM Education , 22 (2), 75–82. Google Scholar
• McClure-Brenchley, K. J., Picardo, K., & Overton-Healy, J. ( 2020 ). Beyond learning: Leveraging undergraduate research into marketable workforce skills . Scholarship and Practice of Undergraduate Research , 3 (3), 28–35. https://doi.org/10.18833/spur/3/3/10 Google Scholar
• Mushtaq, I., & Khan, S. ( 2012 ). Factors affecting students’ academic performance . Global Journal of Management and Business Research , 12 (9), 17–22. Google Scholar
• Peteroy-Kelly, M. A., Marcello, M. R., Crispo, E., Buraei, Z., Strahs, D., Isaacson, M., Jaworski, L., Lopatto, D., & Zuzga, D. ( 2017 ). Participation in a year-long CURE embedded into major core genetics and cellular and molecular biology laboratory courses results in gains in foundational biological concepts and experimental design skills by novice undergraduate researchers . Journal of Microbiology & Biology Education , 18 (1), 18.1.10.  https://doi.org/10.1128/jmbe.v18i1.1226 Google Scholar
• Russell, S. H., Hancock, M. P., & McCullough, J. ( 2007 ). THE PIPELINE: Benefits of undergraduate research experiences . Science , 316 (5824), 548–549.  https://doi.org/10.1126/science.1140384 Medline ,  Google Scholar
• Shadle, S. E., Marker, A., & Earl, B. ( 2017 ). Faculty drivers and barriers: Laying the groundwork for undergraduate STEM education reform in academic departments . International Journal of STEM Education , 4 (1), 8.  https://doi.org/10.1186/s40594-017-0062-7 Medline ,  Google Scholar
• Shannon, C. E. ( 1948 ). A mathematical theory of communication . Bell System Technical Journal , 27 (3), 379–423.  https://doi.org/10.1002/j.1538-7305.1948.tb01338.x Google Scholar
• Theobald, R., & Freeman, S. ( 2014 ) Is it the intervention or the students? Using linear regression to control for student characteristics in undergraduate STEM education research . CBE—Life Sciences Education , 13 (1), 41–48.  https://doi.org/10.1187/cbe-13-07-0136 Link ,  Google Scholar
• Ul-Huda, S., Ali, T., Cassum, S., Nanji, K., & Yousafzai, J. ( 2018 ). Faculty perception about active learning strategies: A cross sectional survey . Journal of Liaquat University of Medical & Health Sciences , 17 (02), 96–100. https://doi.org/10.22442/jlumhs.18172055817 Google Scholar
• Van Lanen, R. J., McGannon, T., & Lockie, N. M. ( 2000 ). Predictors of nursing students’ performance in a one-semester organic and biochemistry course . Journal of Chemical Education , 77 (6), 767.  https://doi.org/10.1021/ed077p767 Google Scholar
• Wolkow, T. D., Durrenberger, L. T., Maynard, M. A., Harrall, K. K., & Hines, L. M. ( 2014 ). A comprehensive faculty, staff, and student training program enhances student perceptions of a course-based research experience at a two-year institution . CBE—Life Sciences Education , 13 (4), 724–737.  https://doi.org/10.1187/cbe.14-03-0056 Link ,  Google Scholar
• Wolkow, T. D., Jenkins, J., Durrenberger, L., Swanson-Hoyle, K., & Hines, L. M. ( 2019 ). One early course-based undergraduate research experience produces sustainable knowledge gains, but only transient perception gains . Journal of Microbiology & Biology Education , 20 (2), 10.  https://doi.org/10.1128/jmbe.v20i2.1679 Google Scholar

Submitted: 23 May 2022 Revised: 9 January 2024 Accepted: 22 July 2024

© 2024 N. T. Appel et al. CBE—Life Sciences Education © 2024 The American Society for Cell Biology. This article is distributed by The American Society for Cell Biology under license from the author(s). It is available to the public under an Attribution–Noncommercial–Share Alike 3.0 Unported Creative Commons License (http://creativecommons.org/licenses/by-nc-sa/3.0).

Deep learning: systematic review, models, challenges, and research directions

• Open access
• Published: 07 September 2023
• Volume 35 , pages 23103–23124, ( 2023 )

Cite this article

You have full access to this open access article

• Tala Talaei Khoei   ORCID: orcid.org/0000-0002-7630-9034 1 ,
• Hadjar Ould Slimane 1 &
• Naima Kaabouch 1  

16k Accesses

30 Citations

4 Altmetric

Explore all metrics

The current development in deep learning is witnessing an exponential transition into automation applications. This automation transition can provide a promising framework for higher performance and lower complexity. This ongoing transition undergoes several rapid changes, resulting in the processing of the data by several studies, while it may lead to time-consuming and costly models. Thus, to address these challenges, several studies have been conducted to investigate deep learning techniques; however, they mostly focused on specific learning approaches, such as supervised deep learning. In addition, these studies did not comprehensively investigate other deep learning techniques, such as deep unsupervised and deep reinforcement learning techniques. Moreover, the majority of these studies neglect to discuss some main methodologies in deep learning, such as transfer learning, federated learning, and online learning. Therefore, motivated by the limitations of the existing studies, this study summarizes the deep learning techniques into supervised, unsupervised, reinforcement, and hybrid learning-based models. In addition to address each category, a brief description of these categories and their models is provided. Some of the critical topics in deep learning, namely, transfer, federated, and online learning models, are explored and discussed in detail. Finally, challenges and future directions are outlined to provide wider outlooks for future researchers.

Similar content being viewed by others

Future and Discussions

Deep reinforcement learning: a survey

Machine Learning Paradigms: Introduction to Deep Learning-Based Technological Applications

Explore related subjects.

• Artificial Intelligence

Avoid common mistakes on your manuscript.

1 Introduction

The main concept of artificial neural networks (ANN) was proposed and introduced as a mathematical model of an artificial neuron in 1943 [ 1 , 2 , 3 ]. In 2006, the concept of deep learning (DL) was proposed as an ANN model with several layers, which has significant learning capacity. In recent years, DL models have seen tremendous progress in addressing and solving challenges, such as anomaly detection, object detection, disease diagnosis, semantic segmentation, social network analysis, and video recommendations [ 4 , 5 , 6 , 7 ].

Several studies have been conducted to discuss and investigate the importance of the DL models in different applications, as illustrated in Table 1 . For instance, the authors of [ 8 ] reviewed supervised, unsupervised, and reinforcement DL-based models. In [ 9 ], the authors outlined DL-based models, platforms, applications, and future directions. Another survey [ 10 ] provided a comprehensive review of the existing models in the literature in different applications, such as natural processing, social network analysis, and audio. In this study, the authors provided a recent advancement in DL applications and elaborated on some of the existing challenges faced by these applications. In [ 11 ], the authors highlighted different DL-based models, such as deep neural networks, convolutional neural networks, recurrent neural networks, and auto-encoders. They also covered their frameworks, benchmarks, and software development requirements. In [ 12 ], the authors discussed the main concepts of deep learning and neural networks. They also provided several applications of DL in a variety of areas.

Other studies covered particular challenges of DL models. For instance, the authors of [ 13 ] explored the importance of class imbalanced dataset on the performance of the DL models as well as the strengths and weaknesses of the methods proposed in the literature for solving class imbalanced data. Another study [ 14 ] explored the challenges that DL faces in the case of data mining, big data, and information processing due to huge volume of data, velocity, and variety. In [ 15 ], the authors analyzed the complexity of DL-based models and provided a review of the existing studies on this topic. In [ 16 ], the authors focused on the activation functions of DL. They introduced these functions as a strategy in DL to transfer nonlinearly separable input into the more linearly separable data by applying a hierarchy of layers, whereas they provided the most common activation functions and their characteristics.

In [ 17 ], the authors outlined the applications of DL in cybersecurity. They provided a comprehensive literature review of DL models in this field and discussed different types of DL models, such as convolutional neural networks, auto-encoders, and generative adversarial networks. They also covered the applications of different attack categories, such as malware, spam, insider threats, network intrusions, false data injection, and malicious in DL. In another study [ 18 ], the authors focused on detecting tiny objects using DL. They analyzed the performance of different DL in detecting these objects. In [ 19 ], the authors reviewed DL models in the building and construction industry-based applications while they discussed several important key factors of using DL models in manufacturing and construction, such as progress monitoring and automation systems. Another study [ 20 ] focused on using different strategies in the domain of artificial intelligence (AI), including DL in smart grids. In such a study, the authors introduced the main AI applications in smart grids while exploring different DL models in depth. In [ 7 ], the authors discussed the current progress of DL in medical areas and gave clear definitions of DL models and their theoretical concepts and architectures. In [ 21 ], the authors analyzed the DL applications in biology, medicine, and engineering domains. They also provided an overview of this field of study and major DL applications and illustrated the main characteristics of several frameworks, including molecular shuttles.

Despite the existing surveys in the field of DL focusing on a comprehensive overview of these techniques in different domains, the increasing amount of these applications and the existing limitations in the current studies motivated us to investigate this topic in depth. In general, the recent studies in the literature mostly discussed specific learning strategies, such as supervised models, while they did not cover different learning strategies and compare them with each other. In addition, the majority of the existing surveys excluded new strategies, such as online learning or federated learning, from their studies. Moreover, these surveys mostly explored specific applications in DL, such as the Internet of Things, smart grid, or constructions; however, this field of study requires formulation and generalization in different applications. In fact, limited information, discussions, and investigations in this domain may lead to prevent any development and progress in DL-based applications. To fill these gaps, this paper provides a comprehensive survey on four types of DL models, namely, supervised, unsupervised, reinforcement, and hybrid learning. It also provides the major DL models in each category and describes the main learning strategies, such as online, transfer, and federated learning. Finally, a detailed discussion of future direction and challenges is provided to support future studies. In short, the main contributions of this paper are as follows:

Classifications and in-depth descriptions of supervised, unsupervised, enforcement, and hybrid models. Description and discussion of learning strategies, such as online, federated, and transfer learning,

Comparison of different classes of learning strategies, their advantages, and disadvantages,

Current challenges and future directions in the domain of deep learning.

The remainder of this paper is organized as follows: Sect.  2 provides descriptions of the supervised, unsupervised, reinforcement, and hybrid learning models, along with a brief description of the models in each category. Section  3 highlights the main learning approaches that are used in deep learning. Section  4 discusses the challenges and future directions in the field of deep learning. The conclusion is summarized in Sect.  5 .

2 Categories of deep learning models

DL models can be classified into four categories, namely, deep supervised, unsupervised, reinforcement learning, and hybrid models. Figure  1 depicts the main categories of DL along with examples of models in each category. In the following, short descriptions of these categories are provided. In addition, Table 2 provides the most common techniques in every category.

Schematic review of the models in deep learning

2.1 Deep supervised learning

Deep supervised learning-based models are one of the main categories of deep learning models that use a labeled training dataset to be trained. These models measure the accuracy through a function, loss function, and adjust the weights till the error has been minimized sufficiently. Among the supervised deep learning category, three important models are identified, namely, deep neural networks, convolutional neural networks, and recurrent neural network-based models, as illustrated in Fig.  2 . Artificial neural networks (ANN), known as neural networks or neural nets, are one of the computing systems, which are inspired by biological neural networks. ANN models are a collection of connected nodes (artificial neurons) that model the neurons in a biological brain. One of the simple ANN models is known as a deep neural network (DNN) [ 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 ]. DNN models consist of a hierarchical architecture with input, output, and hidden layers, each of which has a nonlinear information processing unit, as illustrated in Fig.  2 A. DNN, using the architecture of neural networks, consists of functions with higher complexity when the number of layers and units in a layer is increased. Some known instances of DNN models, as highlighted in Table 2 , are multi-layer perceptron, shallow neural network, operational neural network, self-operational neural network, and iterative residual blocks neural network.

Inner architecture of deep supervised models

The second type of deep supervised models is convolutional neural networks (CNN), known as one of the important DL models that are used to capture the semantic correlations of underlying spatial features among slice-wise representations by convolution operations in multi-dimensional data [ 25 ]. A simple architecture of CNN-based models is shown in Fig.  2 B. In these models, the feature mapping has k filters that are partitioned spatially into several channels. In addition, the pooling function can shrink the width and height of the feature map, while the convolutional layer can apply a filter to an input to generate a feature map that can summarize the identified features as input. The convolutional layers are followed by one or more fully connected layers connected to all the neurons of the previous layer. CNN usually analyzes the hidden patterns using pooling layers for scaling functions, sharing the weights for reducing memories, and filtering the semantic correlation captured by convolutional operations. Therefore, CNN architecture provides a strong potential in spatial features. However, CNN models suffer from their disability in capturing particular features. Some known examples of this network are presented in Table 2 [ 7 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 ].

The other type of supervised DL is recurrent neural network (RNN) models, which are designed for sequential time-series data where the output is returned to the input, as shown in Fig.  2 C [ 27 ]. RNN-based models are widely used to memorize the previous inputs and handle the sequential data and existing inputs [ 42 ]. In RNN models, the recursive process has hidden layers with loops that indicate effective information about the previous states. In traditional neural networks, the given inputs and outputs are totally independent of one another, whereas the recurrent layers of RNN have a memory that remembers the whole data about what is exactly calculated [ 48 ]. In fact, in RNN, similar parameters for every input are applied to construct the neural network and estimate the outputs. The critical principle of RNN-based models is to model time collection samples; hence, specific patterns can be estimated to be dependent on previous ones [ 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 ]. Table 2 provides the instances of RNN-based models as simple recurrent neural network, long short-term memory, gated recurrent unit neural network, bidirectional gated recurrent unit neural network, bidirectional long short-term memory, and residual gated recurrent neural network [ 64 , 65 , 66 ]. Table  3 shows the advantages and disadvantages of supervised DL models.

3 Deep unsupervised learning

Deep unsupervised models have gained significant interest as a mainstream of viable deep learning models. These models are widely used to generate systems that can be trained with few numbers of unlabeled samples [ 24 ]. The models can be classified into auto-encoders , restricted Boltzmann machine, deep belief neural networks, and generative adversarial networks. An auto-encoder (AE) is a type of auto-associative feed-forward neural network that can learn effective representations from the given input in an unsupervised manner [ 29 ]. Figure  3 A provides a basic architecture of AE. As it can be seen, there are three elements in AE, encoder, latent space, and decoder. Initially, the corresponding input passes through the encoder. The encoder is mostly a fully connected ANN that is able to generate the code. In contrast, the decoder generates the outputs using the codes and has an architecture similar to ANN. The aim of having an encoder and decoder is to present an identical output with the given input. It is notable that the dimensionality of the input and output has to be similar. Additionally, real-world data usually suffer from redundancy and high dimensionality, resulting in lower computational efficiency and hindering the modeling of the representation. Thus, a latent space can address this issue by representing compressed data and learning the features of the data, and facilitating data representations to find patterns. As shown in Table 2 , AE consists of several known models, namely, stacked, variational, and convolutional AEs [ 30 , 43 ]. The advantages and disadvantages of these models are presented in Table 4 . 

Inner architecture of deep unsupervised models

The restricted Boltzmann machine (RBM) model, known as Gibbs distribution, is a network of neurons that are connected to each other, as shown in Fig.  3 B. In RBM, the network consists of two layers, namely, the input or visible layer and the hidden layer. There is no output layer in RBM, while the Boltzmann machines are random and generative neural networks that can solve combinative problems. Some common RBM are presented in Table 2 as shallow restricted Boltzmann machines and convolutional restricted Boltzmann machines. The deep belief network (DBN) is another unsupervised deep neural network that performs in a similar way as the deep feed-forward neural network with inputs and multiple computational layers, known as hidden layers, as illustrated in Fig.  3 C. In DBM, there are two main phases that are necessary to be performed, pre-train and fine-tuning phases. The pre-train phase consists of several hidden layers; however, fine-tuning phase only is considered a feed-forward neural network to train and classify the data. In addition, DBN has multiple layers with values, while there is a relation between layers but not with the values [ 31 ]. Table 2 reviews some of the known DBN models, namely, shallow deep belief neural networks and conditional deep belief neural networks [ 44 , 45 ].

The generative adversarial network (GAN) is an another type of unsupervised deep learning model that uses a generator network (GN) and discriminator network (DN) to generate synthetic data to follow similar distribution from the original data, as presented in Fig.  3 D. In this context, the GN mimics the distribution of the given data using noise vectors to exhaust the DN to classify between fake and real samples. The DN can be trained to differentiate between fake and real samples by the GN from the original samples. In general, the GN learns to create plausible data, whereas the DN can learn to identify the generator’s fake data from the real ones. Additionally, the discriminator can penalize the generator for generating implausible data [ 32 , 54 ]. The known types of GAN are presented in Table 2 as generative adversarial networks, signal augmented self-taught learning, and Wasserstein generative adversarial networks. As a result of this discussion, Table 4 provides the main advantages and disadvantages of the unsupervised DL categories [ 56 ].

3.1 Deep reinforcement learning

Reinforcement learning (RL) is the science of making decisions with learning the optimal behavior in an environment to achieve maximum reward. The optimal behavior is achieved through interactions with the environment. In RL, an agent can make decisions, monitor the results, and adjust its technique to provide optimal policy [ 75 , 76 ]. In particular, RL is applied to assist an agent in learning the optimal policy when the agent has no information about the surrounding environments. Initially, the agent monitors the current state, takes action, and receives its reward with its new state. In this context, the immediate reward and new state can adjust the agent's policy; This process is repeated till the agent’s policy is getting close to the optimal policy. To be precise, RL does not need any detailed mathematical model for the system to guarantee optimal control [ 77 ]; however, the agent considers the target system as the environment and optimizes the control policy by communicating with it. The agent performs specific steps. During every step, the agent selects an action based on its existing policy, and the environment feeds back a reward and goes to the next state [ 78 , 79 , 80 ]. This process is learned by the agent to adjust its policy by referencing the relationships during the state, action, and rewards. The RL agent also can determine an optimal policy related to the maximum cumulative reward. In addition, an RL agent can be modeled as Markov decision process (MDP) [ 78 ]. In MDP, when the states and action spaces are finite, the process is known as finite. As it is clear, the RL learning approach may take a huge amount of time to achieve the best policy and discover the knowledge of a whole system; hence, RL is inappropriate for large-scale networks [ 81 ].

In the past few years, deep reinforcement learning (DRL) was proposed as an advanced model of RL in which DL is applied as an effective tool to enhance the learning rate for RL models. The achieved experiences are stored during the real-time learning process, whereas the generated data for training and validating neural networks are applied [ 82 ]. In this context, the trained neural network has to be used to assist the agent in making optimal decisions in real-time scenarios. DRL overcomes the main shortcomings of RL, such as long processing time to achieve optimal policy, thus opening a new horizon to embrace the DRL [ 83 ]. In general, as shown in Fig.  4 , DRL uses the deep neural networks’ characteristics to train the learning process, resulting in increasing the speed and improving the algorithms’ performance. In DRL, within the environment or agent interactions, the deep neural networks keep the internal policy of the agent, which indicates the next action according to the current state of the environment.

Inner architecture of deep reinforcement learning

DRL can be divided into three methods, value-based, policy-based, and model-based methods. Value-based DRL mainly represents and finds the value functions and their optimal ones. In such methods, the agent learns the state or state-action value and behaves based on the best action in the state. One necessary step of these methods is to explore the environment. Some known instances of value-based DRL are deep Q-learning, double deep Q-learning, and duel deep Q-learning [ 83 , 84 , 85 ]. On the contrary, policy-based DRL finds an optimal policy, stochastic or deterministic, to better convergence on high-dimensional or continuous action space. These methods are mainly optimization techniques in which the maximum policy of function can be found. Some examples of policy-based DRL are deep deterministic policy gradient and asynchronous advantage actor critic [ 86 ]. The third category of DRL, model-based methods, aims at learning the functionality of the environment and its dynamics from its previous observations, while these methods attempt a solution using the specific model. For these methods, in the case of having a model, they find the best policy to be efficient, while the process may fail when the state space is huge. In model-based DRL, the model is often updated, and the process is replanned. Instances of model-based DRL are imagination-augmented agents, model-based priors for model-free, and model-based value expansion. Table 5 illustrates the important advantages and disadvantages of these categories [ 87 , 88 , 89 ].

3.2 Hybrid deep learning

Deep learning models have weaknesses and strengths in terms of hyperparameter tuning settings and data explorations [ 45 ]. Therefore, the highlighted weakness of these models can hinder them from being strong techniques in different applications. Every DL model also has characteristics that make it efficient for specific applications; hence, to overcome these shortcomings, hybrid DL models have been proposed based on individual DL models to tackle the shortcomings of specific applications [ 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 , 87 , 88 , 89 ]. Figure  5 indicates the popular hybrid DL models that are used in the literature. It is observed that convolutional neural networks and recurrent neural networks are widely used in existing studies and have high applicability and potentiality compared to other developed DL models.

Review of popular hybrid models

4 Evaluation metrics

In any classification tasks, the metrics are required to evaluate the DL models. It is worth mentioning that various metrics can be used in different fields of studies. It means that the metrics which are used in medical analysis are mostly different with other domains, such as cybersecurity or computer visions. For this reason, we provide a short descriptions and a mathematical equations of the most common metrics in different domains, as following:

Accuracy: It is mainly used in classification problems to indicate the correct predictions made by a DL model. This metric is calculated, as shown in Eq. ( 1 ), where \({T}_{\mathrm{P}}\) is the true positive, \({T}_{\mathrm{N}}\) is true negative, \({F}_{\mathrm{P}}\) is the false positive, and \({F}_{\mathrm{N}}\) is the false negative.

Precision: It refers to the number of the true positives divided by the total number of the positive predictions, including true positive and false positive. This metric can be measured as following:

Recall (detection rate): It measures the number of the positive samples that are classified correctly to the total number of the positive samples. This metric, as measuring in Eq. ( 3 ), can indicate the model’s ability to classify positive samples among other samples.

F1-Score: It is calculated from the precision and recall of the test, where the precision is defined as Eq. ( 2 ), and recall is presented in Eq. ( 3 ). This metric is calculated as shown in Eq. ( 4 ):

Area under the receiver operating characteristics curve (AUC): AUC is one of the important metrics in classification problems. Receiver operating characteristic (ROC) helps to visualize the tradeoff between sensitivity and specificity in DL models. The AUC curve is a plot of true-positive rate (TPR) to false-positive rate (FPR). A good DL model has an AUC value near to 1. This metric is measured, as shown in Eq. ( 5 ), where x is the varying AUC parameter.

False Alarm Rate: This metric is also known as false-positive rate, which is the probability of a false alarm will be raised. It means, a positive result will be given when a true value is negative. This metric can be measured as shown in Eq. ( 6 ):

Misdetection Rate: It is a metric that shows the percentage of misclassified samples. This metric can be defined as the percentage of the samples that are not detected. It is also measured, as shown in Eq. ( 7 ):

5 Learning classification in deep learning models

Learning strategies, as shown in Fig.  6 , include online learning, transfer learning, and federated learning. In this section, these learning strategies are discussed in brief.

Review of learning classification in deep learning models

5.1 Online learning

Conventional machine learning models mostly employ batch learning methods, in which a collection of training data is provided in advance to the model. This learning method requires the whole training dataset to be made accessible ahead to the training, which lead to high memory usage and poor scalability. On the other hand, online learning is a machine learning category where data are processed in sequential order, and the model is updated accordingly [ 90 ]. The purpose of online learning is to maximize the accuracy of the prediction model using the ground truth of previous predictions [ 91 ]. Unlike batch or offline machine learning approaches, which require the complete training dataset to be available to be trained on [ 92 ], online learning models use sequential stream of data to update their parameters after each data instance. Online learning is mainly optimal when the entire dataset is unavailable or the environment is dynamically changing [ 92 , 93 , 94 , 95 , 96 ]. On the other hand, batch learning is easier to maintain and less complex; it requires all the data to be available to be trained on it and does not update its model. Table 6 shows the advantages and disadvantages of batch learning and online learning.

An online model aims to learn a hypothesis \({\mathcal{H}}:X \to Y\) Where \(X\) is the input space, and \(Y\) is the output space. At each time step \(t\) , a new data instance \({\varvec{x}}_{{\varvec{t}}} \in X\) is received, and an output or prediction \(\hat{y}_{t}\) is generated using the mapping function \({\mathcal{H}}\left( {x_{t} ,w_{t} } \right) = \hat{y}_{t}\) , where \({{\varvec{w}}}_{{\varvec{t}}}\) is the weights’ vector of the online model at the time step \(t\) . The true class label \({y}_{t}\) is then utilized to calculate the loss and update the weights of the model \({\varvec{w}}_{{{\varvec{t}} + 1}}\) , which is illustrated in Fig.  7 [ 97 ].

Online machine learning process

The number of mistakes committed by the online model across T time steps is defined as \({M}_{T}\) for \(\hat{y}_{t} \ne y_{t}\) [ 55 ]. The goal of an online learning model is to minimize the total loss of the online model performance compared to the best model in hindsight, which is defined as [ 35 ]

where the first term is the sum of the loss function at time step t, and the second term is the loss function of the best model after seeing all the instances [ 98 , 99 ]. While training the online model, different approaches can be adopted regarding data that the model has already trained on; full memory, in which the model preserves all training data instances; partial memory, where the model retains only some of the training data instances; and no memory, in which it remembers none. Two main techniques are utilized to remove training data instances: passive forgetting and active forgetting [ 107 , 108 , 109 ]

Passive forgetting only considers the amount of time that has passed since the training data instances were received by the model, which implies that the significance of data diminishes over time.

Active forgetting , on the other hand, requires additional information from the utilized training data in order to determine which objects to remove. The density-based forgetting and error-based forgetting are two active forgetting techniques.

Online learning techniques can be classified into three categories: online learning with full feedback, online learning with partial feedback, and online learning with no feedback. Online learning with full feedback is when all training data instances \(x\) have a corresponding true label \(y\) which is always disclosed to the model at the end of each online learning round. Online learning with partial feedback is when only partial feedback information is received that shows if the prediction is correct or not, rather than the corresponding true label explicitly. In this category, the online learning model is required to make online updates by seeking to maintain a balance between the exploitation of revealed knowledge and the exploration of unknown information with the environment [ 2 ]. On the other hand, online learning with no feedback is when only the training data are fed to the model without the ground truth or feedback. This category includes online clustering and dimension reduction [ 99 , 100 , 101 , 102 , 103 , 104 , 105 , 106 , 107 , 108 , 109 , 110 , 111 ].

5.2 Deep transfer learning

Training deep learning models from scratch needs extensive computational and memory resources and large amounts of labeled datasets. However, for some types of scenarios, huge, annotated datasets are not always available. Additionally, developing such datasets requires a great deal of time and is a costly operation. Transfer learning (TL) has been proposed as an alternative for training deep learning models [ 112 ]. In TL, the obtained knowledge from another domain can be easily transferred to target another classification problem. TL saves computing resources and increases efficiency in training new deep learning models. TL can also help train deep learning models on available annotated datasets before validating them on unlabeled data [ 113 , 114 ]. Figure  8 illustrates a simple visualization of the deep transfer learning, which can transfer valuable knowledge by further using the learning ability of neural networks.

Visualization of deep transfer learning

In this survey, the deep transfer learning techniques are classified based on the generalization viewpoints between deep learning models and domains into four categories, namely, instance, feature representation, model parameter, and relational knowledge-based techniques. In the following, we briefly discuss these categories with their categorizations, as illustrated in Fig.  9 .

Categories of deep transfer learning

5.2.1 Instance-based

Instance-based TL techniques are performed based on the selected instance or on selecting different weights for instances. In such techniques, the TL aims at training a more accurate model under a transfer scenario, in which the difference between a source and a target comes from the different marginal probability distributions or conditional probability distributions [ 62 ]. Instance-based TL presents the labeled samples that are only limited to training a classification model in the target domain. This technique can directly margin the source data into the target data, resulting in decreasing the target model performance and a negative transfer during training [ 109 , 110 , 111 ]. The main goal of instance-based TL is to single out the instances in the source domains. Such a process can have positive impact on the training of the models in target as well as augmenting the target data through particular weighting techniques. In this context, a viable solution is to learn the weights of the source domains' instances automatically in an objective function. The objective function is given by:

where \({W}_{i}\) is the weighting coefficient of the given source instance, \({C}^{s}\) represents the risks function of the selected source instance, and \({\vartheta }^{*}\) is the second risk function related to the target task or the parameter regularization.

The weighting coefficient of the given source instance can be computed as the ratio of the marginal probability distribution between source and target domains. Instance-based TL can be categorized into two subcategories, weight estimation and heuristic re-weighting-based techniques [ 63 ]. A weight estimation method can focus on scenarios in which there are limited labeled instances in the target domain, converting the instance transfer problem into the weight estimation problem using kernel embedding techniques. In contrast, a heuristic re-weighting technique is more effective for developing deep TL tasks that have labeled instances and are available in the target domains [ 64 ]. This technique aims at detecting negative source instances by applying instance re-weighting approaches in a heuristic manner. One of the known instance re-weighting approaches is the transfer adaptive boosting algorithm, in which the weights of the source and target instances are updated via several iterations [ 116 ].

5.2.2 Feature representation-based

Feature representation-based TL models can share or learn a common feature representation between a target and a source domain. This category uses models with the ability to transfer knowledge by learning similar representations at the feature space level. Its main aim is to learn the mapping function as a bridge to transfer raw data in source and target domains from various feature spaces to a latent feature space [ 109 ]. From a general perspective, feature representation-based TL covers two transfer styles with or without adapting to the target domain [ 110 ]. Techniques without adapting to the target domain can extract representations as inputs for the target models; however, the techniques with adapting to the target domain can extract feature representations across various domains via domain adaption techniques [ 112 ]. In general, techniques of adapting to the target domain are hard to implement, and their assumptions are weak to be justified in most of cases. On the contrary, techniques of adapting to the target domain are easy to implement, and their assumptions can be strong in different scenarios [ 111 ].

One important challenge in feature representation TL with domain adaptation is the estimation of representing invariance between source and target domains. There are three techniques to build representation invariance, leveraging discrepancy-based, adversarial-based, and reconstruction-based. Leveraging discrepancy-based can improve the learning transferable ability representations and decrease the discrepancy based on distance metrics between a given source and target, while the adversarial-based is inspired by GANs and provides the neural network with the ability to learn domain-invariant representations. In construction-based, the auto-encoder neural networks with specific task classifiers are combined to optimize the encoder architecture, which takes domain-specific representations and shares an encoder that learns representations between different domains [ 113 ].

5.2.3 Model parameter-based

Model parameter-based TL can share the neural network architecture and parameters between target and source domains. This category can convey the assumptions that can share in common between the source and target domains. In such a technique, transferable knowledge is embedded into the pre-trained source model. This pre-trained source model has a particular architecture with some parameters in the target model [ 99 ]. The aim of this process is to use a section of the pre-trained model in the source domain, which can improve the learning process in the target domain. These techniques are performed based on the assumption that labeled instances in the target domain are available during the training of the target model [ 99 , 100 , 101 , 102 , 103 ]. Model parameter-based TL is divided into two categories, sequential and joint training. In sequential training, the target deep model can be established by pretraining a model on an auxiliary domain. However, joint training focuses on developing the source and target tasks at the same time. There are two methods to perform joint training [ 104 ]. The first method is hard parameter sharing, which shares the hidden layers directly while maintaining the task-specific layers independently [ 99 , 100 , 101 , 102 , 103 , 104 , 105 , 106 , 107 , 108 , 109 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 ]. The second method is soft parameter sharing which changes the weight coefficient of the source and target tasks and adds regularization to the risk function. Table 7 shows the advantages and disadvantages of the three categories, instance-based, future representation-based, and model parameter-based.

5.3 Deep federated learning

In traditional centralized DL, the collected data have to be stored on local devices, such as personal computers [ 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 , 87 ]. In general, traditional centralized DL can store the user data on the central server and apply it for training and testing purposes, as illustrated in Fig.  10 A, while this process may deal with several shortcomings, such as high computational power, low security, and privacy. In such models, the efficiency and accuracy of the models heavily depend on the computational power and training process of the given data on a centralized server. As a result, centralized DL models not only provide low privacy and high risks of data leakage but also indicate the high demands on storage and computing capacities of the several machines which train the models in parallel. Therefore, federated learning (FL) was proposed as an emerging technology to address such challenges [ 104 , 105 , 106 , 107 , 108 , 109 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 , 119 ].

Centralized and federated learning process flow

FL provides solutions to keep the users’ privacy by decentralizing data from the corresponding central server to devices and enabling artificial intelligence (AI) methods to discipline the data. Figure  10 B summarizes the main process in an FL model. In particular, the unavailability of sufficient data, high computational power, and a limited level of privacy using local data are three major benefits of FL AI over centralized AI [ 115 , 116 , 117 , 118 , 119 ]. For this purpose, FL models aim at training a global model which can be trained on data distributed on several devices while they can protect the data. In this context, FL finds an optimal global model, known as \(\theta\) , can minimize the aggregated local loss function, \({f}_{k}\) ( \({\theta }^{k}\) ), as shown in Eq. ( 10 ).

where X denotes the data feature, y is the data label, \({n}_{k}\) is the local data size, C is the ratio in which the local clients do not participate in every round of the models’ updates, l is the loss function, k is the client index, and \(\sum_{k=1}^{C*k}{n}_{k}\) shows the total number of sample pairs. FL can be classified based on the characteristics of the data distribution among the clients into two types, namely, vertical and horizontal FL models, as discussed in the following:

5.3.1 Horizontal federated learning

Horizontal FL, homogeneous FL, shows the cases in which the given training data of the participating clients share a similar feature space; however, these corresponding data have various sample spaces [ 76 ]. Client one and Client two have several data rows with similar features, whereas each row shows specific data for a unique client. A typical common algorithm, namely, federated averaging (FedAvg), is usually used as a horizontal FL algorithm. FedAvg is one of the most efficient algorithms for distributing training data with multiple clients. In such an algorithm, clients keep the data local for protecting their privacy, while central parameters are applied to communicate between different clients [ 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 , 87 , 88 , 89 , 90 , 91 , 92 , 93 , 94 , 95 , 96 , 97 , 98 , 99 , 100 , 101 , 102 , 103 , 104 , 105 , 106 , 107 , 108 , 109 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 , 119 , 120 , 121 , 122 ].

In addition, horizontal FL provides efficient solutions to avoid leaking private local data. This can happen since the global and local model parameters are only permitted to communicate between the servers and clients, whereas all the given training data are stored on the client devices without being accessed by any other parties [ 14 , 119 , 120 , 121 , 122 , 123 , 124 , 125 , 126 , 127 , 128 , 129 , 130 , 131 , 132 , 133 ]. Despite such advantages, constant downloading and uploading in horizontal FL may consume huge amounts of communication resources. In deep learning models, the situation is getting worse due to the needing huge amounts of computation and memory resources. To address such issues, several studies have been performed to decrease the computational efficiency of horizontal FL models [ 134 ]. These studies proposed methods to reduce communication costs using multi-objective evolutionary algorithms, model quantization, and sub-sampling techniques. In these studies, however, no private data can be accessed directly by any third party, the uploaded model parameters or gradients may still leak the data for every client [ 135 ].

5.3.2 Vertical federated learning

Vertical FL, heterogeneous FL, is one of the types of FL in which users’ training data can share the same sample space while they have multiple different feature spaces. Client one and Client two have similar data samples with different feature spaces, and all clients have their own local data that are mostly assumed to one client keeps all the data classes. Such clients with data labels are known as guest parties or active parties, and clients without labels are known as host parties [ 136 ]. In particular, in vertical FL, the common data between unrelated domains are mainly applied to train global DL models [ 137 ]. In this context, participants may use intermediate third-party resources to indicate encryption logic to guarantee the data stats are kept. Although it is not necessary to use third parties in this process, studies have demonstrated that vertical FL models with third parties using encryption techniques provide more acceptable results [ 14 , 89 , 90 , 91 , 92 , 93 , 94 , 95 , 96 , 97 , 98 , 99 , 100 , 101 , 102 , 103 , 104 , 105 , 106 , 107 , 108 , 109 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 , 119 , 120 , 121 , 122 , 123 , 124 , 125 , 126 , 127 , 128 , 129 , 130 , 131 , 132 , 133 , 134 , 135 , 136 , 137 , 138 ].

In contrast with horizontal FL, training parametric models in vertical FL has two benefits. Firstly, trained models in vertical FL have a similar performance as centralized models. As a matter of fact, the computed loss function in vertical FL is the same as the loss function in centralized models. Secondly, vertical FL often consumes fewer communication resources compared to horizontal FL [ 138 ]. Vertical FL only consumes more communication resources than horizontal FL if and only if the data size is huge. In vertical FL, privacy preservation is the main challenge. For this purpose, several studies have been conducted to investigate privacy preservation in vertical FL, using identity resolution schemes, protocols, and vertical decision learning schemes. Although these approaches improve the vertical FL models, there are still some main slight differences between horizontal and vertical FL [ 100 , 101 , 102 , 103 , 104 , 105 , 106 , 107 , 108 , 109 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 , 119 , 120 , 121 , 122 , 123 , 124 , 125 , 126 , 127 , 128 , 129 , 130 , 131 , 132 , 133 , 134 , 135 , 136 , 137 , 138 , 139 , 140 , 141 , 142 , 143 ].

Horizontal FL includes a server for aggregation of the global models. In contrast, the vertical FL does not have a central server and global model [ 14 , 122 , 123 , 124 , 125 , 126 , 127 , 128 , 129 , 130 ]. As a result, the output of the local model’s aggregation is done based on the guest client to build a proper loss function. Another difference is the model parameters or gradients between servers and clients in horizontal FL. Local model parameters in vertical FL depend on the local data feature spaces, while the guest client receives model outputs from the connected host clients [ 143 ]. In this process, the intermediate gradient values are sent back for updating local models [ 105 ]. Ultimately, the server and the clients communicate with one another once in a communication round in horizontal FL; however, the guest and host clients have to send and receive data several times in a communication round in vertical FL [ 14 , 106 , 107 , 108 , 109 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 , 119 , 120 , 121 , 122 , 123 , 124 , 125 , 126 , 127 , 128 ]. Table 8 summarizes the main advantages and disadvantages of vertical and horizontal FL and compares these FL learning categories with central learning.

6 Challenges and future directions

Deep learning models, while powerful and versatile, face several significant challenges. Addressing these challenges requires a multidisciplinary approach involving data collection and preprocessing techniques, algorithmic enhancements, fairness-aware model training, interpretability methods, safe learning, robust models to adversarial attacks, and collaboration with domain experts and affected communities to push the boundaries of deep learning and realize its full potential. A brief description of each of these challenges is given below.

6.1 Data availability and quality

Deep learning models require large amounts of labeled training data to learn effectively. However, obtaining sufficient and high-quality labeled data can be expensive, time-consuming, or challenging, particularly in specialized domains or when dealing with sensitive data such cybersecurity. Although there are several approaches, such as data augmentation, to generate high amounts of data, it can sometimes be cumbersome to generate enough training data and satisfy the requirements of DL models. In addition, having a small dataset may lead to overfitting issues where DL models perform well on the training data but fail to generalize to unseen data. Balancing model complexity and regularization techniques to avoid overfitting while achieving good generalization is a challenge in deep learning. In addition, exploring techniques to improve data efficiency, such as few-shot learning, active learning, or semi-supervised learning, remains an active area of research.

6.2 Ethics and fairness

The challenge of ethics and fairness in deep learning underscores the critical need to address biases, discrimination, and social implications embedded within these models. Deep learning systems learn patterns from vast and potentially biased datasets, which can perpetuate and amplify societal prejudices, leading to unfair or unjust outcomes. The ethical dilemma lies in the potential for these models to unintentionally marginalize certain groups or reinforce systemic disparities. As deep learning is increasingly integrated into decision-making processes across domains such as hiring, lending, and criminal justice, ensuring fairness and transparency becomes paramount. Striving for ethical deep learning involves not only detecting and mitigating biases but also establishing guidelines and standards that prioritize equitable treatment, encompassing a multidisciplinary effort to foster responsible AI innovation for the betterment of society.

6.3 Interpretability and explainability

Interpretability and explainability of deep learning pose significant challenges in understanding the inner workings of complex models. As deep neural networks become more intricate, with numerous layers and parameters, their decision-making processes often resemble “black boxes,” making it difficult to discern how and why specific predictions are made. This lack of transparency hinders the trust and adoption of these models, especially in high-stakes applications like health care and finance. Striking a balance between model performance and comprehensibility is crucial to ensure that stakeholders, including researchers, regulators, and end-users, can gain meaningful insights into the model's reasoning, enabling informed decisions and accountability while navigating the intricate landscape of modern deep learning.

6.4 Robustness to adversarial attacks

Deep learning models are susceptible to adversarial attacks, a concerning vulnerability that highlights the fragility of their decision boundaries. Adversarial attacks involve making small, carefully crafted perturbations to input data, often imperceptible to humans, which can lead to misclassification or erroneous outputs from the model. These attacks exploit the model's sensitivity to subtle changes in its input space, revealing a lack of robustness in real-world scenarios. Adversarial attacks not only challenge the reliability of deep learning systems in critical applications such as autonomous vehicles and security systems but also underscore the need for developing advanced defense mechanisms and more resilient models that can withstand these intentional manipulations. Therefore, developing robust models that can withstand such attacks and maintaining model security and data is of high importance.

6.5 Catastrophic forgetting

Catastrophic forgetting, or catastrophic interference, is a phenomenon that can occur in online deep learning, where a model forgets or loses previously learned information when it learns new information. This can lead to a degradation in performance on tasks that were previously well-learned as the model adjusts to new data. This catastrophic forgetting is particularly problematic because deep neural networks often have a large number of parameters and complex representations. When a neural network is trained on new data, the optimization process may adjust the weights and connections in a way that erases the knowledge the network had about previous tasks. Therefore, there is a need for models that address this phenomenon.

6.6 Safe learning

Safe deep learning models are designed and trained with a focus on ensuring safety, reliability, and robustness. These models are built to minimize risks associated with uncertainty, hazards, errors, and other potential failures that can arise in the deployment and operation of artificial intelligence systems. DL models without safety and risks considerations in ground or aerial robots can lead to unsafe outcomes, serious damage, and even casualties. The safety properties include estimating risks, dealing with uncertainty in data, and detecting abnormal system behaviors and unforeseen events to ensure safety and avoid catastrophic failures and hazards. The research in this area is still at a very early stage.

6.7 Transfer learning and adaptation

Transfer learning and adaptation present complex challenges in the realm of deep learning. While pretraining models on large datasets can capture valuable features and representations, effectively transferring this knowledge to new tasks or domains requires overcoming hurdles related to differences in data distributions, semantic gaps, and contextual variations. Adapting pre-trained models to specific target tasks demands careful fine-tuning, domain adaptation, or designing novel architectures that can accommodate varying input modalities and semantics. The challenge lies in striking a balance between leveraging the knowledge gained from pretraining and tailoring the model to extract meaningful insights from the new data, ensuring that the transferred representations are both relevant and accurate. Successfully addressing the intricacies of transfer learning and adaptation in deep learning holds the key to unlocking the full potential of AI across diverse applications and domains.

7 Conclusions

In recent years, deep learning has emerged as a prominent data-driven approach across diverse fields. Its significance lies in its capacity to reshape entire industries and tackle complex problems that were once challenging or insurmountable. While numerous surveys have been published on deep learning, its models, and applications, a notable proportion of these surveys has predominantly focused on supervised techniques and their potential use cases. In contrast, there has been a relative lack of emphasis on deep unsupervised and deep reinforcement learning methods. Motivated by these gaps, this survey offers a comprehensive exploration of key learning paradigms, encompassing supervised, unsupervised, reinforcement, and hybrid learning, while also describing prominent models within each category. Furthermore, it delves into cutting-edge facets of deep learning, including transfer learning, online learning, and federated learning. The survey finishes by outlining critical challenges and charting prospective pathways, thereby illuminating forthcoming research trends across diverse domains.

Data availability

Not applicable.

Tan C, Sun F, Kong T, Zhang W, Yang C, Liu C (2018) A survey on deep transfer learning. In: International conference on artificial neural networks, Springer, Berlin; p 270–279

Tang B, Chen Z, Hefferman G, Pei S, Wei T, He H, Yang Q (2017) Incorporating intelligence in fog computing for big data analysis in smart cities. IEEE Trans Ind Informatics 13:2140–2150

Google Scholar  

Khoei TT, Aissou G, Al Shamaileh K, Devabhaktuni VK, Kaabouch N (2023) Supervised deep learning models for detecting GPS spoofing attacks on unmanned aerial vehicles. In: 2023 IEEE international conference on electro information technology (eIT), Romeoville, IL, USA, pp 340–346. https://doi.org/10.1109/eIT57321.2023.10187274

Article   Google Scholar  

Nguyen TT, Nguyen QVH, Nguyen DT, Nguyen DT, Huynh-The T, Nahavandi S, Nguyen TT, Pham QV, Nguyen CM (2022) Deep learning for deepfakes creation and detection: a survey. Comput Vis Image Underst 223:103525

Dong S, Wang P, Abbas K (2021) A survey on deep learning and its applications. Comput Scie Rev 40:100379

MathSciNet   MATH   Google Scholar  

Ni J, Young T, Pandelea V, Xue F, Cambria E (2022) Recent advances in deep learning based dialogue systems: a systematic survey. Artif Intell Rev 56:1–101

Piccialli F, Di Somma V, Giampaolo F, Cuomo S, Fortino G (2021) A survey on deep learning in medicine: why, how and when? Inf Fus 66:111–137

Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117

Hatcher WG, Yu W (2018) A survey of deep learning: platforms, applications and emerging research trends. IEEE Access 6:24411–24432. https://doi.org/10.1109/ACCESS.2018.2830661

Pouyanfar S, Sadiq S, Yan Y, Tian H, Tao Y, Reyes MP, Shyu ML, Chen SC, Iyengar SS (2018) A survey on deep learning: algorithms, techniques, and applications. ACM Comput Surv (CSUR) 51(5):1–36

Alom MZ et al (2019) A state-of-the-art survey on deep learning theory and architectures. Electronics 8(3):292. https://doi.org/10.3390/electronics8030292

Dargan S, Kumar M, Ayyagari MR, Kumar G (2020) A survey of deep learning and its applications: a new paradigm to machine learning. Arch of Computat Methods Eng 27(4):1071–1092

MathSciNet   Google Scholar  

Johnson JM, Khoshgoftaar TM (2019) Survey on deep learning with class imbalance. J Big Data 6(1):1–54

Zhang Q, Yang LT, Chen Z, Li P (2018) A survey on deep learning for big data. Inf Fus 42:146–157

Hu X, Chu L, Pei J, Liu W, Bian J (2021) Model complexity of deep learning: a survey. Knowl Inf Syst 63(10):2585–2619

Dubey SR, Singh SK, Chaudhuri BB (2022) Activation functions in deep learning: a comprehensive survey and benchmark. Neurocomputing 503:92–108

Berman D, Buczak A, Chavis J, Corbett C (2019) A survey of deep learning methods for cyber security. Information 10(4):122. https://doi.org/10.3390/info10040122

Tong K, Wu Y (2022) Deep learning-based detection from the perspective of small or tiny objects: a survey. Image Vis Comput 123:104471

Baduge SK, Thilakarathna S, Perera JS, Arashpour M, Sharafi P, Teodosio B, Shringi A, Mendis P (2022) Artificial intelligence and smart vision for building and construction 4.0: machine and deep learning methods and applications. Autom Constr 141:104440

Omitaomu OA, Niu H (2021) Artificial intelligence techniques in smart grid: a survey. Smart Cities 4(2):548–568. https://doi.org/10.3390/smartcities4020029

Akay A, Hess H (2019) Deep learning: current and emerging applications in medicine and technology. IEEE J Biomed Health Inform 23(3):906–920. https://doi.org/10.1109/JBHI.2019.2894713

Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi FE (2017) A survey of deep neural network architectures and their applications. Neurocomputing 234:11–26

Srinidhi CL, Ciga O, Martel AL (2021) Deep neural network models for computational histopathology: a survey. Med Image Anal 67:101813

Kattenborn T, Leitloff J, Schiefer F, Hinz S (2021) Review on convolutional neural networks (CNN) in vegetation remote sensing. ISPRS J Photogramm Remote Sens 173:24–49

Tugrul B, Elfatimi E, Eryigit R (2022) Convolutional neural networks in detection of plant leaf diseases: a review. Agriculture 12(8):1192

Yadav SP, Zaidi S, Mishra A, Yadav V (2022) Survey on machine learning in speech emotion recognition and vision systems using a recurrent neural network (RNN). Arch Computat Methods Eng 29(3):1753–1770

Mai HT, Lieu QX, Kang J, Lee J (2022) A novel deep unsupervised learning-based framework for optimization of truss structures. Eng Comput 39:1–24

Jiang H, Peng M, Zhong Y, Xie H, Hao Z, Lin J, Ma X, Hu X (2022) A survey on deep learning-based change detection from high-resolution remote sensing images. Remote Sens 14(7):1552

Mousavi SM, Beroza GC (2022) Deep-learning seismology. Science 377(6607):eabm4470

Song X, Li J, Cai T, Yang S, Yang T, Liu C (2022) A survey on deep learning based knowledge tracing. Knowl-Based Syst 258:110036

Wang J, Biljecki F (2022) Unsupervised machine learning in urban studies: a systematic review of applications. Cities 129:103925

Li Y (2022) Research and application of deep learning in image recognition. In: 2022 IEEE 2nd international conference on power, electronics and computer applications (ICPECA), p 994–999

Borowiec ML, Dikow RB, Frandsen PB, McKeeken A, Valentini G, White AE (2022) Deep learning as a tool for ecology and evolution. Methods Ecol Evol 13(8):1640–1660

Wang X et al (2022) Deep reinforcement learning: a survey. IEEE Trans on Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2022.3207346

Pateria S, Subagdja B, Tan AH, Quek C (2021) Hierarchical reinforcement learning: A comprehensive survey. ACM Comput Surv (CSUR) 54(5):1–35

Amroune M (2019) Machine learning techniques applied to on-line voltage stability assessment: a review. Arch Comput Methods Eng 28:273–287

Liu S, Shi R, Huang Y, Li X, Li Z, Wang L, Mao D, Liu L, Liao S, Zhang M et al (2021) A data-driven and data-based framework for online voltage stability assessment using partial mutual information and iterated random forest. Energies 14:715

Ahmad A, Saraswat D, El Gamal A (2023) A survey on using deep learning techniques for plant disease diagnosis and recommendations for development of appropriate tools. Smart Agric Technol 3:100083

Khan A, Khan SH, Saif M, Batool A, Sohail A, Waleed Khan M (2023) A survey of deep learning techniques for the analysis of COVID-19 and their usability for detecting omicron. J Exp Theor Artif Intell. https://doi.org/10.1080/0952813X.2023.2165724

Wang C, Gong L, Wang A, Li X, Hung PCK, Xuehai Z (2017) SOLAR: services-oriented deep learning architectures. IEEE Trans Services Comput 14(1):262–273

Moshayedi AJ, Roy AS, Kolahdooz A, Shuxin Y (2022) Deep learning application pros and cons over algorithm deep learning application pros and cons over algorithm. EAI Endorsed Trans AI Robotics 1(1):1–13

Huang L, Luo R, Liu X, Hao X (2022) Spectral imaging with deep learning. Light: Sci Appl 11(1):61

Bhangale KB, Kothandaraman M (2022) Survey of deep learning paradigms for speech processing. Wireless Pers Commun 125(2):1913–1949

Khojaste-Sarakhsi M, Haghighi SS, Ghomi SF, Marchiori E (2022) Deep learning for Alzheimer’s disease diagnosis: a survey. Artif Intell Med 130:102332

Fu G, Jin Y, Sun S, Yuan Z, Butler D (2022) The role of deep learning in urban water management: a critical review. Water Res 223:118973

Kim L-W (2018) DeepX: deep learning accelerator for restricted Boltzmann machine artificial neural networks. IEEE Trans Neural Netw Learn Syst 29(5):1441–1453

Wang C, Gong L, Yu Q, Li X, Xie Y, Zhou X (2017) DLAU: a scalable deep learning accelerator unit on FPGA. IEEE Trans Comput-Aided Design Integr Circuits Syst 36(3):513–517

Dundar A, Jin J, Martini B, Culurciello E (2017) Embedded streaming deep neural networks accelerator with applications. IEEE Trans Neural Netw Learn Syst 28(7):1572–1583

De Mauro A, Greco M, Grimaldi M, Nobili G (2016) Beyond data scientists: a review of big data skills and job families. In: Proceedings of IFKAD, p 1844–1857

Lin S-B (2019) Generalization and expressivity for deep nets. IEEE Trans Neural Netw Learn Syst 30(5):1392–1406

Gopinath M, Sethuraman SC (2023) A comprehensive survey on deep learning based malware detection techniques. Comp Sci Rev 47:100529

MATH   Google Scholar  

Khalifa NE, Loey M, Mirjalili S (2022) A comprehensive survey of recent trends in deep learning for digital images augmentation. Artif Intell Rev 55:1–27

Peng S, Cao L, Zhou Y, Ouyang Z, Yang A, Li X, Jia W, Yu S (2022) A survey on deep learning for textual emotion analysis in social networks. Digital Commun Netw 8(5):745–762

Tao X, Gong X, Zhang X, Yan S, Adak C (2022) Deep learning for unsupervised anomaly localization in industrial images: a survey. IEEE Trans Instrum Meas 71:1–21. https://doi.org/10.1109/TIM.2022.3196436

Sharifani K, Amini M (2023) Machine learning and deep learning: a review of methods and applications. World Inf Technol Eng J 10(07):3897–3904

Li Q, Peng H, Li J, Xia C, Yang R, Sun L, Yu PS, He L (2022) A survey on text classification: from traditional to deep learning. ACM Trans Intell Syst Technol (TIST) 13(2):1–41

Zhou Z, Xiang Y, Hao Xu, Yi Z, Shi Di, Wang Z (2021) A novel transfer learning-based intelligent nonintrusive load-monitoring with limited measurements. IEEE Trans Instrum Meas 70:1–8

Akram MW, Li G, Jin Y, Chen X, Zhu C, Ahmad A (2020) Automatic detection of photovoltaic module defects in infrared images with isolated and develop-model transfer deep learning. Sol Energy 198:175–186

Karimipour H, Dehghantanha A, Parizi RM, Choo K-KR, Leung H (2019) A deep and scalable unsupervised machine learning system for cyber-attack detection in large-scale smart grids. IEEE Access 7:80778–80788

Moonesar IA, Dass R (2021) Artificial intelligence in health policy—a global perspective. Global J Comput Sci Technol 1:1–7

Mo Y, Wu Y, Yang X, Liu F, Liao Y (2022) Review the state-of-the-art technologies of semantic segmentation based on deep learning. Neurocomputing 493:626–646

Subramanian N, Elharrouss O, Al-Maadeed S, Chowdhury M (2022) A review of deep learning-based detection methods for COVID-19. Comput Biol Med 143:105233

Tsuneki M (2022) Deep learning models in medical image analysis. J Oral Biosci 64(3):312–320

Pan X, Lin X, Cao D, Zeng X, Yu PS, He L, Nussinov R, Cheng F (2022) Deep learning for drug repurposing: Methods, databases, and applications. Wiley Interdiscip Rev: Computat Mol Sci 12(4):e1597

Novakovsky G, Dexter N, Libbrecht MW, Wasserman WW, Mostafavi S (2023) Obtaining genetics insights from deep learning via explainable artificial intelligence. Nat Rev Genet 24(2):125–137

Fan Y, Tao B, Zheng Y, Jang S-S (2020) A data-driven soft sensor based on multilayer perceptron neural network with a double LASSO approach. IEEE Trans Instrum Meas 69(7):3972–3979

Menghani G (2023) Efficient deep learning: a survey on making deep learning models smaller, faster, and better. ACM Comput Surv 55(12):1–37

Mehrish A, Majumder N, Bharadwaj R, Mihalcea R, Poria S (2023) A review of deep learning techniques for speech processing. Inf Fus 99:101869

Mohammed A, Kora R (2023) A comprehensive review on ensemble deep learning: opportunities and challenges. J King Saud Univ-Comput Inf Sci 35:757–774

Alzubaidi L, Bai J, Al-Sabaawi A, Santamaría J, Albahri AS, Al-dabbagh BSN, Fadhel MA, Manoufali M, Zhang J, Al-Timemy AH, Duan Y (2023) A survey on deep learning tools dealing with data scarcity: definitions, challenges, solutions, tips, and applications. J Big Data 10(1):46

Katsogiannis-Meimarakis G, Koutrika G (2023) A survey on deep learning approaches for text-to-SQL. The VLDB J. https://doi.org/10.1007/s00778-022-00776-8

Soori M, Arezoo B, Dastres R (2023) Artificial intelligence, machine learning and deep learning in advanced robotics a review. Cognitive Robotics 3:57–70

Mijwil M, Salem IE, Ismaeel MM (2023) The significance of machine learning and deep learning techniques in cybersecurity: a comprehensive review. Iraqi J Comput Sci Math 4(1):87–101

de Oliveira RA, Bollen MH (2023) Deep learning for power quality. Electr Power Syst Res 214:108887

Yin L, Gao Qi, Zhao L, Zhang B, Wang T, Li S, Liu H (2020) A review of machine learning for new generation smart dispatch in power systems. Eng Appl Artif Intell 88:103372

Luong NC et al. (2019) Applications of deep reinforcement learning in communications and networking: a survey. In: IEEE communications surveys & tutorials, vol 21, no 4, p 3133–3174, https://doi.org/10.1109/COMST.2019.2916583

Kiran BR et al (2022) Deep reinforcement learning for autonomous driving: a survey. IEEE Trans Intell Transp Syst 23(6):4909–4926. https://doi.org/10.1109/TITS.2021.3054625

Arulkumaran K, Deisenroth MP, Brundage M, Bharath AA (2017) Deep Reinforcement Learning: A Brief Survey. IEEE Signal Process Mag 34(6):26–38. https://doi.org/10.1109/MSP.2017.2743240

Levine S, Kumar A, Tucker G, Fu J (2020) Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643

Vinuesa R, Azizpour H, Leite I, Balaam M, Dignum V, Domisch S, Felländer A, Langhans SD, Tegmark M, Nerini FF (2020) The role of artificial intelligence in achieving the sustainable development goals. Nature Commun. https://doi.org/10.1038/s41467-019-14108-y

Khoei TT, Kaabouch N (2023) ACapsule Q-learning based reinforcement model for intrusion detection system on smart grid. In: 2023 IEEE international conference on electro information technology (eIT), Romeoville, IL, USA, pp 333–339. https://doi.org/10.1109/eIT57321.2023.10187374

Hoi SC, Sahoo D, Lu J, Zhao P (2021) Online learning: a comprehensive survey. Neurocomputing 459:249–289

Celard P, Iglesias EL, Sorribes-Fdez JM, Romero R, Vieira AS, Borrajo L (2023) A survey on deep learning applied to medical images: from simple artificial neural networks to generative models. Neural Comput Appl 35(3):2291–2323

Mohammad-Rahimi H, Rokhshad R, Bencharit S, Krois J, Schwendicke F (2023) Deep learning: a primer for dentists and dental researchers. J Dent 130:104430

Liu Z, Tong L, Chen L, Jiang Z, Zhou F, Zhang Q, Zhang X, Jin Y, Zhou H (2023) Deep learning based brain tumor segmentation: a survey. Complex Intell Syst 9(1):1001–1026

Zheng Y, Xu Z, Xiao A (2023) Deep learning in economics: a systematic and critical review. Artif Intell Rev 4:1–43

Jia T, Kapelan Z, de Vries R, Vriend P, Peereboom EC, Okkerman I, Taormina R (2023) Deep learning for detecting macroplastic litter in water bodies: a review. Water Res 231:119632

Newbury R, Gu M, Chumbley L, Mousavian A, Eppner C, Leitner J, Bohg J, Morales A, Asfour T, Kragic D, Fox D (2023) Deep learning approaches to grasp synthesis: a review. IEEE Trans Robotics. https://doi.org/10.1109/TRO.2023.3280597

Shafay M, Ahmad RW, Salah K, Yaqoob I, Jayaraman R, Omar M (2023) Blockchain for deep learning: review and open challenges. Clust Comput 26(1):197–221

Benczúr AA., Kocsis L, Pálovics R (2018) Online machine learning in big data streams. arXiv preprint arXiv:1802.05872

Shalev-Shwartz S (2011) Online learning and online convex optimization. Found Trends® Mach Learn 4(2):107–194

Millán Giraldo M, Sánchez Garreta JS (2008) A comparative study of simple online learning strategies for streaming data. WSEAS Trans Circuits Syst 7(10):900–910

Pinto G, Wang Z, Roy A, Hong T, Capozzoli A (2022) Transfer learning for smart buildings: a critical review of algorithms, applications, and future perspectives. Adv Appl Energy 5:100084

Sayed AN, Himeur Y, Bensaali F (2022) Deep and transfer learning for building occupancy detection: a review and comparative analysis. Eng Appl Artif Intell 115:105254

Li C, Zhang S, Qin Y, Estupinan E (2020) A systematic review of deep transfer learning for machinery fault diagnosis. Neurocomputing 407:121–135

Li W, Huang R, Li J, Liao Y, Chen Z, He G, Yan R, Gryllias K (2022) A perspective survey on deep transfer learning for fault diagnosis in industrial scenarios: theories, applications and challenges. Mech Syst Signal Process 167:108487

Wan Z, Yang R, Huang M, Zeng N, Liu X (2021) A review on transfer learning in EEG signal analysis. Neurocomputing 421:1–14

Tan C, Sun F, Kong T (2018) A survey on deep transfer learning.In: Proceedings of international conference on artificial neural networks. p 270–279

Qian F, Gao W, Yang Y, Yu D et al (2020) Potential analysis of the transfer learning model in short and medium-term forecasting of building HVAC energy consumption. Energy 193:116724

Weber M, Doblander C, Mandl P, (2020b). Towards the detection of building occupancy with synthetic environmental data. arXiv preprint arXiv:2010.04209

Zhu H, Xu J, Liu S, Jin Y (2021) Federated learning on non-IID data: a survey. Neurocomputing 465:371–390

Ouadrhiri AE, Abdelhadi A (2022) Differential privacy for deep and federated learning: a survey. IEEE Access 10:22359–22380. https://doi.org/10.1109/ACCESS.2022.3151670

Zhang C, Xie Y, Bai H, Yu B, Li W, Gao Y (2021) A survey on federated learning. Knowl-Based Syst 216:106775

Banabilah S, Aloqaily M, Alsayed E, Malik N, Jararweh Y (2022) Federated learning review: fundamentals, enabling technologies, and future applications. Inf Process Manag 59(6):103061

Mothukuri V, Parizi RM, Pouriyeh S, Huang Y, Dehghantanha A, Srivastava G (2021) A survey on security and privacy of federated learning. Futur Gener Comput Syst 115:619–640

McMahan HB, Moore E, Ramage D, Hampson S, Arcas BA (2017) Communication-efficient learning of deep networks from decentralized data. In: Proceedings of the 20th international conference on artificial intelligence and statistics, AISTATS

Hardy S, Henecka W, Ivey-Law H, Nock R, Patrini G, Smith G, Thorne B (2017) Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption. arXiv preprint arXiv:1711.10677

Chen T, He T, Benesty M, Khotilovich V, Tang Y, Cho H et al (2015) Xgboost: extreme gradient boosting. R Package Vers 1:4–2

Heng K, Fan T, Jin Y, Liu Y, Chen T, Yang Q (2019) Secureboost: a lossless federated learning framework. arXiv preprint arXiv:1901.08755

Konečný J, McMahan HB, Yu FX, Richtárik P, Suresh AT, Bacon D (2016) Federated learning: strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492

Hamedani L, Liu R, Atat J, Wu Y (2017) Reservoir computing meets smart grids: attack detection using delayed feedback networks. IEEE Trans Industr Inf 14(2):734–743

Yuan X, Xie L, Abouelenien M (2018) A regularized ensemble framework of deep learning for cancer detection from multi-class, imbalanced training data. Pattern Recogn 77:160–172

Xiao B, Xiong J, Shi Y (2016) Novel applications of deep learning hidden features for adaptive testing. In: Proceedings of the 21st Asia and South Pacifc design automation conference, p 743–748

Zhong SH, Li Y, Le B (2015) Query oriented unsupervised multi document summarization via deep learning. Expert Syst Appl 42:1–10

Vincent P et al (2010) Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J Mach Learn Res 11:3371–3408

Alom MZ et al. (2017) Object recognition using cellular simultaneous recurrent networks and convolutional neural network. In: Neural networks (IJCNN), international joint conference on IEEE

Quang W, Stokes JW (2016) MtNet: a multi-task neural network for dynamic malware classification. in: proceedings of the international conference detection of intrusions and malware, and vulnerability assessment, Donostia-San Sebastián, Spain, 7–8 July, p 399–418

Kamilaris A, Prenafeta-Boldú FX (2018) Deep learning in agriculture: a survey. Comput Electron Agric 147:70–90

Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, Van Der Laak JA, Van Ginneken B, Sánchez CI (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88

Gheisari M, Ebrahimzadeh F, Rahimi M, Moazzamigodarzi M, Liu Y, Dutta Pramanik PK, Heravi MA, Mehbodniya A, Ghaderzadeh M, Feylizadeh MR, Kosari S (2023) Deep learning: applications, architectures, models, tools, and frameworks: a comprehensive survey. CAAI Trans Intell Technol. https://doi.org/10.1049/cit2.12180

Pichler M, Hartig F (2023) Machine learning and deep learning—a review for ecologists. Methods Ecol Evolut 14(4):994–1016

Wang N, Chen T, Liu S, Wang R, Karimi HR, Lin Y (2023) Deep learning-based visual detection of marine organisms: a survey. Neurocomputing 532:1–32

Lee M (2023) The geometry of feature space in deep learning models: a holistic perspective and comprehensive review. Mathematics 11(10):2375

Xu M, Yoon S, Fuentes A, Park DS (2023) A comprehensive survey of image augmentation techniques for deep learning. Pattern Recogn 137:109347

Minaee S, Abdolrashidi A, Su H, Bennamoun M, Zhang D (2023) Biometrics recognition using deep learning: a survey. Artif Intell Rev 56:1–49

Xiang H, Zou Q, Nawaz MA, Huang X, Zhang F, Yu H (2023) Deep learning for image inpainting: a survey. Pattern Recogn 134:109046

Chakraborty S, Mali K (2022) An overview of biomedical image analysis from the deep learning perspective. Research anthology on improving medical imaging techniques for analysis and intervention. IGI Global, Hershey, pp 43–59

Lestari, N.I., Hussain, W., Merigo, J.M. and Bekhit, M., 2023, January. A Survey of Trendy Financial Sector Applications of Machine and Deep Learning. In: Application of big data, blockchain, and internet of things for education informatization: second EAI international conference, BigIoT-EDU 2022, Virtual Event, July 29–31, 2022, Proceedings, Part III, Springer Nature, Cham, p. 619–633

Chaddad A, Peng J, Xu J, Bouridane A (2023) Survey of explainable AI techniques in healthcare. Sensors 23(2):634

Grumiaux PA, Kitić S, Girin L, Guérin A (2022) A survey of sound source localization with deep learning methods. J Acoust Soc Am 152(1):107–151

Zaidi SSA, Ansari MS, Aslam A, Kanwal N, Asghar M, Lee B (2022) A survey of modern deep learning based object detection models. Digital Signal Process 126:103514

Dong J, Zhao M, Liu Y, Su Y, Zeng X (2022) Deep learning in retrosynthesis planning: datasets, models and tools. Brief Bioinf 23(1):391

Zhan ZH, Li JY, Zhang J (2022) Evolutionary deep learning: a survey. Neurocomputing 483:42–58

Matsubara Y, Levorato M, Restuccia F (2022) Split computing and early exiting for deep learning applications: survey and research challenges. ACM Comput Surv 55(5):1–30

Zhang B, Rong Y, Yong R, Qin D, Li M, Zou G, Pan J (2022) Deep learning for air pollutant concentration prediction: a review. Atmos Environ 290:119347

Yu X, Zhou Q, Wang S, Zhang YD (2022) A systematic survey of deep learning in breast cancer. Int J Intell Syst 37(1):152–216

Behrad F, Abadeh MS (2022) An overview of deep learning methods for multimodal medical data mining. Expert Syst Appl 200:117006

Mittal S, Srivastava S, Jayanth JP (2022) A survey of deep learning techniques for underwater image classification. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2022.3143887

Tercan H, Meisen T (2022) Machine learning and deep learning based predictive quality in manufacturing: a systematic review. J Intell Manuf 33(7):1879–1905

Stefanini M, Cornia M, Baraldi L, Cascianelli S, Fiameni G, Cucchiara R (2022) From show to tell: a survey on deep learning-based image captioning. IEEE Trans Pattern Anal Mach Intell 45(1):539–559

Caldas S, Konečný J, McMahan HB, Talwalkar A (2018) Expanding the reach of federated learning by reducing client resource requirements. arXiv preprint arXiv:1812.07210

Chen Y, Sun X, Jin Y (2019) Communication-efficient federated deep learning with layerwise asynchronous model update and temporally weighted aggregation. IEEE Trans Neural Netw Learn Syst 31:4229–4238

Zhu H, Jin Y (2019) Multi-objective evolutionary federated learning. IEEE Trans Neural Netw Learn Syst 31:1310–1322

Download references

Acknowledgements

The authors acknowledge the support of the National Science Foundation (NSF), Award Number 2006674.

Author information

Authors and affiliations.

School of Electrical Engineering and Computer Science, University of North Dakota, Grand Forks, ND, 58202, USA

Tala Talaei Khoei, Hadjar Ould Slimane & Naima Kaabouch

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Tala Talaei Khoei .

Ethics declarations

Conflict of interest.

The authors declare no conflicts of interest relevant to this article.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Talaei Khoei, T., Ould Slimane, H. & Kaabouch, N. Deep learning: systematic review, models, challenges, and research directions. Neural Comput & Applic 35 , 23103–23124 (2023). https://doi.org/10.1007/s00521-023-08957-4

Download citation

Received : 31 May 2023

Accepted : 15 August 2023

Published : 07 September 2023

Issue Date : November 2023

DOI : https://doi.org/10.1007/s00521-023-08957-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Artificial intelligence
• Neural networks
• Deep learning
• Supervised learning
• Unsupervised learning
• Reinforcement learning
• Online learning
• Federated learning
• Transfer learning
• Find a journal
• Publish with us
• Track your research

deep learning Recently Published Documents

Total documents.

• Latest Documents
• Most Cited Documents
• Contributed Authors
• Related Sources
• Related Keywords

Synergic Deep Learning for Smart Health Diagnosis of COVID-19 for Connected Living and Smart Cities

COVID-19 pandemic has led to a significant loss of global deaths, economical status, and so on. To prevent and control COVID-19, a range of smart, complex, spatially heterogeneous, control solutions, and strategies have been conducted. Earlier classification of 2019 novel coronavirus disease (COVID-19) is needed to cure and control the disease. It results in a requirement of secondary diagnosis models, since no precise automated toolkits exist. The latest finding attained using radiological imaging techniques highlighted that the images hold noticeable details regarding the COVID-19 virus. The application of recent artificial intelligence (AI) and deep learning (DL) approaches integrated to radiological images finds useful to accurately detect the disease. This article introduces a new synergic deep learning (SDL)-based smart health diagnosis of COVID-19 using Chest X-Ray Images. The SDL makes use of dual deep convolutional neural networks (DCNNs) and involves a mutual learning process from one another. Particularly, the representation of images learned by both DCNNs is provided as the input of a synergic network, which has a fully connected structure and predicts whether the pair of input images come under the identical class. Besides, the proposed SDL model involves a fuzzy bilateral filtering (FBF) model to pre-process the input image. The integration of FBL and SDL resulted in the effective classification of COVID-19. To investigate the classifier outcome of the SDL model, a detailed set of simulations takes place and ensures the effective performance of the FBF-SDL model over the compared methods.

A deep learning approach for remote heart rate estimation

Weakly supervised spatial deep learning for earth image segmentation based on imperfect polyline labels.

In recent years, deep learning has achieved tremendous success in image segmentation for computer vision applications. The performance of these models heavily relies on the availability of large-scale high-quality training labels (e.g., PASCAL VOC 2012). Unfortunately, such large-scale high-quality training data are often unavailable in many real-world spatial or spatiotemporal problems in earth science and remote sensing (e.g., mapping the nationwide river streams for water resource management). Although extensive efforts have been made to reduce the reliance on labeled data (e.g., semi-supervised or unsupervised learning, few-shot learning), the complex nature of geographic data such as spatial heterogeneity still requires sufficient training labels when transferring a pre-trained model from one region to another. On the other hand, it is often much easier to collect lower-quality training labels with imperfect alignment with earth imagery pixels (e.g., through interpreting coarse imagery by non-expert volunteers). However, directly training a deep neural network on imperfect labels with geometric annotation errors could significantly impact model performance. Existing research that overcomes imperfect training labels either focuses on errors in label class semantics or characterizes label location errors at the pixel level. These methods do not fully incorporate the geometric properties of label location errors in the vector representation. To fill the gap, this article proposes a weakly supervised learning framework to simultaneously update deep learning model parameters and infer hidden true vector label locations. Specifically, we model label location errors in the vector representation to partially reserve geometric properties (e.g., spatial contiguity within line segments). Evaluations on real-world datasets in the National Hydrography Dataset (NHD) refinement application illustrate that the proposed framework outperforms baseline methods in classification accuracy.

Prediction of Failure Categories in Plastic Extrusion Process with Deep Learning

Hyperparameters tuning of faster r-cnn deep learning transfer for persistent object detection in radar images, a comparative study of automated legal text classification using random forests and deep learning, a semi-supervised deep learning approach for vessel trajectory classification based on ais data, an improved approach towards more robust deep learning models for chemical kinetics, power system transient security assessment based on deep learning considering partial observability, a multi-attention collaborative deep learning approach for blood pressure prediction.

We develop a deep learning model based on Long Short-term Memory (LSTM) to predict blood pressure based on a unique data set collected from physical examination centers capturing comprehensive multi-year physical examination and lab results. In the Multi-attention Collaborative Deep Learning model (MAC-LSTM) we developed for this type of data, we incorporate three types of attention to generate more explainable and accurate results. In addition, we leverage information from similar users to enhance the predictive power of the model due to the challenges with short examination history. Our model significantly reduces predictive errors compared to several state-of-the-art baseline models. Experimental results not only demonstrate our model’s superiority but also provide us with new insights about factors influencing blood pressure. Our data is collected in a natural setting instead of a setting designed specifically to study blood pressure, and the physical examination items used to predict blood pressure are common items included in regular physical examinations for all the users. Therefore, our blood pressure prediction results can be easily used in an alert system for patients and doctors to plan prevention or intervention. The same approach can be used to predict other health-related indexes such as BMI.

Export Citation Format

Share document.

• Data Science
• Data Analysis
• Data Visualization
• Machine Learning
• Deep Learning
• Computer Vision
• Artificial Intelligence
• AI ML DS Interview Series
• AI ML DS Projects series
• Data Engineering
• Web Scrapping

10 Must Read Machine Learning Research Papers

Machine learning is a rapidly evolving field with research papers often serving as the foundation for discoveries and advancements. For anyone keen to delve into the theoretical and practical aspects of machine learning, the following ten research papers are essential reads. They cover foundational concepts, groundbreaking techniques, and key advancements in the field.

Table of Content

1. “A Few Useful Things to Know About Machine Learning” by Pedro Domingos

2. “imagenet classification with deep convolutional neural networks” by alex krizhevsky, ilya sutskever, and geoffrey e. hinton, 3. “playing atari with deep reinforcement learning” by volodymyr mnih et al., 4. “sequence to sequence learning with neural networks” by ilya sutskever, oriol vinyals, and quoc v. le, 5. “attention is all you need” by ashish vaswani et al., 6. “generative adversarial nets” by ian goodfellow et al., 7. “bert: pre-training of deep bidirectional transformers for language understanding” by jacob devlin et al., 8. “deep residual learning for image recognition” by kaiming he et al., 9. “a survey on deep learning in medical image analysis” by geert litjens et al., 10. “alphago: mastering the game of go with deep neural networks and tree search” by silver et al..

This article highlights 10 must-read machine learning research papers that have significantly contributed to the development and understanding of machine learning. Whether you’re a beginner or an experienced practitioner, these papers provide invaluable insights that will help you grasp the complexities of machine learning and its potential to transform industries.

Summary : Pedro Domingos provides a comprehensive overview of essential machine learning concepts and common pitfalls. This paper is a great starting point for understanding the broader landscape of machine learning.

Key Contributions:

• Distills core principles and practical advice.
• Discusses overfitting, feature engineering, and model selection.
• Offers insights into the trade-offs between different machine learning algorithms.

Access: Read the Paper

Summary : Often referred to as the “AlexNet” paper, this work introduced a deep convolutional neural network that significantly improved image classification benchmarks, marking a turning point in computer vision.

• Demonstrated the power of deep learning for image classification.
• Introduced techniques like dropout and ReLU activations.
• Showed the importance of large-scale datasets and GPU acceleration.

Summary : This paper from DeepMind presents the use of deep Q-networks (DQN) to play Atari games . It was a seminal work in applying deep learning to reinforcement learning.

• Introduced the concept of using deep learning for Q-learning.
• Showcased the ability of DQNs to learn complex behaviors from raw pixel data.
• Paved the way for further research in reinforcement learning.

Summary : This paper introduced the sequence-to-sequence (seq2seq) learning framework , which has become fundamental for tasks such as machine translation and text summarization.

• Proposed an encoder-decoder architecture for sequence tasks.
• Demonstrated effective training of neural networks for sequence modeling.
• Laid the groundwork for subsequent advancements in natural language processing.

Summary : This paper introduces the Transformer model, which relies solely on attention mechanisms, discarding recurrent layers used in previous models. It has become the backbone of many modern NLP systems.

• Proposed the Transformer architecture, which uses self-attention to capture dependencies.
• Demonstrated improvements in training efficiency and performance over RNN-based models.
• Led to the development of models like BERT, GPT, and others.

Summary : Ian Goodfellow and his colleagues introduced Generative Adversarial Networks (GANs) , a revolutionary framework for generating realistic data through adversarial training.

• Proposed a novel approach where two neural networks compete against each other.
• Enabled the generation of high-quality images, text, and other data types.
• Spurred a plethora of research on GAN variations and applications.

Summary : BERT (Bidirectional Encoder Representations from Transformers) introduced a new way of pre-training language models, significantly improving performance on various NLP benchmarks.

• Proposed bidirectional training of transformers to capture context from both directions.
• Achieved state-of-the-art results on several NLP tasks.
• Set the stage for subsequent models like RoBERTa, ALBERT, and DistilBERT.

Summary : This paper introduces Residual Networks (ResNets), which utilize residual learning to train very deep neural networks effectively.

• Addressed the issue of vanishing gradients in very deep networks.
• Demonstrated that extremely deep networks can be trained successfully.
• Improved performance on image classification tasks and influenced subsequent network architectures.

Summary : This survey provides a comprehensive review of deep learning techniques applied to medical image analysis, summarizing the state of the art in this specialized field.

• Reviewed various deep learning methods used in medical imaging.
• Discussed challenges and future directions in the field.
• Provided insights into applications such as disease detection and image segmentation.

Summary : This paper describes AlphaGo, the first AI to defeat a world champion in the game of Go, using a combination of deep neural networks and Monte Carlo tree search.

• Demonstrated the effectiveness of combining deep learning with traditional search techniques.
• Achieved a major milestone in AI by mastering a complex game.
• Influenced research in AI and its application to other complex decision-making problems.

These ten research papers cover a broad spectrum of machine learning advancements, from foundational concepts to cutting-edge techniques. They provide valuable insights into the development and application of machine learning technologies, making them essential reads for anyone looking to deepen their understanding of the field. By exploring these papers, you can gain a comprehensive view of how machine learning has evolved and where it might be heading in the future.

10 Must Read Machine Learning Research Papers – FAQ’s

What are large language models (llms) and why are they important.

Large Language Models (LLMs) are advanced AI systems designed to understand and generate human language. They are built using deep learning techniques, particularly transformer architectures. LLMs are important because they enable applications such as text generation, translation, and sentiment analysis, significantly advancing the field of natural language processing (NLP).

Why should I read “A Few Useful Things to Know About Machine Learning” by Pedro Domingos?

Pedro Domingos’ paper provides a broad overview of key machine learning concepts, common challenges, and practical advice. It’s an excellent resource for both beginners and experienced practitioners to understand the underlying principles of machine learning and avoid common pitfalls.

What impact did “ImageNet Classification with Deep Convolutional Neural Networks” have on the field?

The “AlexNet” paper revolutionized image classification by demonstrating the effectiveness of deep convolutional neural networks. It significantly improved benchmark results on ImageNet and introduced techniques like dropout and ReLU activations, which are now standard in deep learning.

Please Login to comment...

Similar reads.

• AI-ML-DS Blogs
• California Lawmakers Pass Bill to Limit AI Replicas
• Best 10 IPTV Service Providers in Germany
• Python 3.13 Releases | Enhanced REPL for Developers
• IPTV Anbieter in Deutschland - Top IPTV Anbieter Abonnements
• Content Improvement League 2024: From Good To A Great Article

Improve your Coding Skills with Practice

What kind of Experience do you want to share?

• Reserve a study room
• Library Account
• Undergraduate Students
• Graduate Students
• Faculty & Staff

Research Lifecycle Online Learning Objects

• Plan and Design
• Collect and Create
• Analyze and Collaborate
• Evaluate and Archive
• Share and Disseminate
• Access and Reuse

Contact Us for Assistance

The health sciences library.

Call toll-free:  (844) 352-7399 E-mail:   Ask Us More contact information

What is the Research Lifecycle?

Within the context of this guide, we will be referring to the research lifecycle as the steps taken to fund, plan, design, deliver, and disseminate a research project from start to finish. For reference, you can also consult the Harvard Biomedical Data Management page to see this process of research data management.   Another useful resource is the SAGE Research Methods Project Planner tool , which walks you through the research process, step-by-step.

This guide compiles library-created online learning objects designed to address a particular stage of the research process. By grouping them here, we hope to provide easy access to faculty and students looking for asynchronous instruction on literature reviews, synthesis tools (e.g., Covidence), data management, finding a publisher, and where to store your research for reuse. 

Resources to Help You Get Started

Related Resources

• C. Kenneth and Dianne Wright Center for Clinical and Translational Research: Consultation Services "Our team of research professionals help with planning, implementing, conducting, and disseminating research. Our services including identifying and accessing resources, matching researchers with collaborators and providing research training."
• Human Research Protection Program/Institutional Review Board "The OVPRI Human Research Protection Program provides guidance and resources for researchers and IRB members at Virginia Commonwealth University. We review all research involving human participants to facilitate the conduct of ethical research and ensure compliance with federal, state and institutional regulations."
• Office of the Vice President for Research and Innovation (OVPRI): Research Development Resources OVPRI provides "a comprehensive range of proposal development and grantsmanship services..."
• Next: Plan and Design >>
• Last Updated: Aug 30, 2024 12:45 PM
• URL: https://guides.library.vcu.edu/researchlifecycle

Northeastern engineer is using AI and cloud computing to empower educational researchers

• Search Search

NSF grant will help Ningfang Mi establish a comprehensive curriculum that incorporates AI, machine learning and cloud computing into educational research training programs.

• Copy Link Link Copied!

Northeastern University professor Ningfang Mi says she can help educational researchers unlock new insights into education policies, teaching strategies and student outcomes by utilizing AI and advanced cyberinfrastructure systems in the cloud.

The first time Mi experienced the educational analysis field, she says, she was surprised to learn that her colleagues in educational research didn’t know about advanced technologies and resources available to them to work with big data. There was a big gap between what educators and educational researchers needed and the tools they had access to or knew how to utilize, she says.

“There’s a very high potential we can help them improve their educational research and get more meaningful insights from their rich datasets,” Mi says.

For years, Mi, a professor of electrical and computer engineering, has been focusing her research on cloud computing — computing services such as servers, storage, databases and networking delivered over the internet to users who don’t have access or don’t want to maintain these IT resources themselves.

Typically, cloud computing, along with technologies like machine learning and artificial intelligence, Mi says, has been used mainly by computer science and engineering researchers, leaving fields like the social sciences behind.

“I want to encourage the development and usage of cyberinfrastructure in other domains and disciplines,” Mi says, “and particularly, in the educational domain.”

To boost adoption of advanced technologies in education, Mi has teamed up with computer science experts from different universities and an educational researcher. Their main goal is to prepare future workers in education analytics to use advanced cyberinfrastructure systems in the cloud. 

“This is not the classic computer-related research,” Mi says. “This is bringing together computer engineering and education research to provide training and resources.”

In the long term, the interdisciplinary team of experts aims to establish a comprehensive curriculum that incorporates AI, machine learning and cloud computing into educational research training programs. This will not only enhance the skills of current and future researchers, Mi says, but also ensure that education policies and practices are informed by the latest advancements in technology and data science.

Featured Stories

Back to School 2024: Students’ guide to essential move-in resources, Welcome Week activities and more

Northeastern graduate revolutionizing the beauty industry with AI-powered solutions for women of color

Everything you need to know about Northeastern convocation ceremonies in Boston, London and Oakland

What I wish I’d known: Northeastern students offer advice to incoming freshmen

Mi and her collaborators have received a National Science Foundation grant for a four-year project titled “AI4EDU: Cloud Infrastructure-Enabled Training for AI in Educational Research and Assessment.” 

AI has become a focal point in educational policy discussions, Mi says, both on federal and state levels. Educational administrators and classroom teachers are being urged to adopt AI-driven tools and techniques, which can significantly enhance their ability to analyze large, complex datasets. 

The researchers believe this interest in AI within the education research community will continue to increase in the coming years, driven by data automatically generated by e-learning systems, online courses and tutoring systems. When combined with AI, this data can unlock new insights into education policies, teaching strategies and student outcomes.

The project will first develop a platform with innovative training modules and materials on the use of cloud computing resources for AI analytics that can be used by education researchers, school administrators, policymakers and even prekindergarten through Grade 12 teachers in North Carolina. Later, the experts plan to add to the platform sample projects with accompanying datasets for real-world hands-on training, data analysis and cloud management tools and a depository where the community can collect and share machine learning programs, datasets and code tailored for a variety of educational research tasks.  

“At the same time, our student researchers can extend their research from image and biomedical data processing with machine learning to educational research,” Mi says.

Some of the research questions that AI in education can help answer, Mi says, include the effects of education policies on teacher effectiveness, student academic achievement and psychological well-being; relationships between school policy, teacher quality, family resources and student academic achievement; and student developmental stages and the appropriate match between teaching strategies and learning styles.

The platform will be open-access, Mi says, meaning educational students and researchers from institutions with limited resources will still be able to benefit from its offerings. 

Mi and her collaborators are striving to make a lasting impact on education.

“This project can lead us to other kinds of multidisciplinary collaboration in the future,” Mi says.

Science & Technology

Recent Stories

• DOI: 10.47119/ijrp1001501620246661
• Corpus ID: 270317421

Supreme Secondary Learner Government and Its Relation in Shaping Center of Learning Culture

• Judie Lorr Maranan Harlan
• Published in International Journal of… 16 May 2024
• Education, Political Science, Sociology

Related Papers

Showing 1 through 3 of 0 Related Papers

Subscribe to our free Newsletter! →

Home › Children News

Your child’s smart toy is a data scientist. Here’s what it’s learning and how it’s watching ‘every move’

By StudyFinds Staff

Reviewed by Steve Fink

Research led by Professor Isabel Wagner, University of Basel

Aug 29, 2024

Twelve toys were examined in a study on smart toys and privacy. (Credit: University of Basel / Céline Emch)

BASEL, Switzerland — Remember when toys were simple? A stuffed animal was just a stuffed animal, and the only data it collected was the occasional ketchup stain or grass mark from outdoor adventures. But in today’s digital age, your child’s favorite playmate might be secretly moonlighting as a miniature surveillance device, collecting data on everything from playtime habits to personal preferences.

Welcome to the brave new world of smart toys, where every playtime could be a potential privacy pitfall. An eye-opening new study by researchers from the University of Basel uncovers alarming shortcomings in the privacy and security features of popular smart toys, raising concerns about the safety of children’s personal information.

The study, published in the journal Privacy Technologies and Policy , examined 12 smart toys available in the European market. These included toys equipped with internet connectivity, microphones, cameras, and the ability to collect and transmit data. They include household names like the Toniebox, tiptoi smart pen and its optional charging station, and the ever-popular Tamagotchi. Think of them as miniature computers disguised as playful companions.

At first glance, these high-tech toys seem like a parent’s dream. Take the Toniebox, for instance. This clever device allows even the youngest children to play their favorite stories and songs with ease – simply place a figurine on the box, and voila! The tale begins. Tilt the box left or right to rewind or fast-forward. It’s so simple that even a toddler can master it .

Here’s where the plot thickens: while your little one is lost in the world of Peppa Pig, the Toniebox is busy creating a digital dossier. According to the study, It meticulously records when it’s activated, which figurine is used when playback stops, and even tracks those rewinds and fast-forwards. All this data is then whisked away to the manufacturer, painting a detailed picture of your child’s play patterns.

The Toniebox isn’t alone in its data-gathering ways . The study found that many smart toys are collecting extensive behavioral data about children, often without clear explanations of how this information will be used or protected. It’s like having a constant surveillance system watching your child’s every move during playtime.

“Children’s privacy requires special protection,” emphasizes Julika Feldbusch, first author of the study, in a statement.

She argues that toy manufacturers should place greater weight on privacy and on the security of their products than they currently do in light of their young target audience. The study also found that most toys lack transparency when it comes to data collection and processing. Privacy policies , when they exist at all, are often vague, difficult to understand, or buried in fine print. This means parents are often in the dark about what information is being collected from their children and how it’s being used.

Security measures were also in need of improvements. While most toys use encryption for data transmitted over the Internet, local network connections – like those used during initial setup – were often unencrypted. Some popular toys, including the Toniebox and tiptoi’s optional charging station were found to have inadequate encryption for data traffic, potentially leaving children’s information vulnerable to interception. The actual tiptoi pen does not record how and when a child uses it. Only audio data for the purchased products is transferred.

In some cases, researchers were able to intercept Wi-Fi passwords and other sensitive information simply by eavesdropping on these local connections.

Perhaps most alarmingly, when the researchers attempted to exercise their rights under Europe’s General Data Protection Regulation (GDPR) by requesting access to the data collected about them, only 43% of toy vendors responded within the legally mandated one-month period. Even then, some of the responses were incomplete or unsatisfactory.

Moreover, many companion apps for these toys were found to request unnecessary and invasive permissions.

“The apps bundled with some of these toys demand entirely unnecessary access rights, such as to a smartphone’s location or microphone,” says Professor Isabel Wagner of the Department of Mathematics and Computer Science at the University of Basel.

In the wrong hands, the data collected by these toys could be used for identity theft , targeted advertising, or even more sinister purposes like child grooming or blackmail. And because children are particularly vulnerable and may not understand the implications of sharing personal information, the onus is on toy manufacturers and parents to ensure their safety in the digital playground.

“We’re already seeing signs of a two-tier society when it comes to privacy protection for children,” says Feldbusch. “Well-informed parents engage with the issue and can choose toys that do not create behavioral profiles of their children. But many lack the technical knowledge or don’t have time to think about this stuff in detail.”

So, what can parents do to protect their tech-savvy tots? The researchers suggest looking for toys that prioritize privacy and security, reading privacy policies carefully (even if they’re boring), and being cautious about granting unnecessary permissions to toy apps. They also recommend that toy makers implement stronger security measures and provide more transparent information about data collection practices.

Prof. Wagner acknowledges that individual children might not experience immediate negative consequences from these data collection practices.

“But nobody really knows that for sure,” she cautions. “For example, constant surveillance can have negative effects on personal development.”

The study serves as a wake-up call not just for parents, but for regulators and toy manufacturers as well. As smart toys become increasingly prevalent, it’s crucial that we find a balance between innovation and protecting our children’s privacy and security.

The researchers hope their work will lead to improved standards and practices in the smart toy industry. In the meantime, parents might want to think carefully about whether that internet-connected teddy bear is really worth the potential risks to their child’s privacy and safety.

Editor’s Note: The University of Basel informed StudyFinds that a portion of their press release did not include important findings regarding the tiptoi pen and its optional charging station. “Not only the tiptoi pen but also its optional charging station was tested in the study. The insufficient encryption of data traffic concerns the optional charging station of the pen,” a university official said in an email to StudyFinds. This has been noted and update in our story.

Ravensburger, the manufacturer of Tiptoi, also contacted StudyFinds regarding the omission and issued the following statement:

“The University of Basel published a press release about a study according to which smart toys – including the tiptoi pen – enable unsafe data traffic in the children’s room and sometimes also collect data about children’s behaviour. These statements are false in the case of tiptoi. What is true, however, is that neither the tiptoi pen nor its charging station collect data or create user profiles. “In the study by the University of Basel, it was not the pen but the optional charging station that was evaluated. This can only be used with the fourth generation of tiptoi pens. The vast majority of customers download their audio files via PC, so the study does not affect most tiptoi owners. “One point of criticism in the study is that some smart toys transfer data unencrypted. However, tiptoi does not transfer any private data, only the audio files and updates for tiptoi products that are publicly available. “The study also criticised the process of setting up the charging station’s internet connection. During this process, which takes a few minutes, the access data is sent unencrypted to the charging station – as is the case with the setup of many other electronic devices. Someone within a radius of 25 metres could theoretically access the access data at this moment. Although the study estimates the risk to be extremely low, Ravensburger is working on offering an alternative transmission method.”

Paper Summary

Methodology.

The researchers employed a comprehensive approach to evaluate 12 diverse smart toys available on Amazon.de, focusing on those with Wi-Fi capabilities. They developed evaluation criteria based on established cybersecurity standards for consumer IoT devices. Their assessment methods included decrypting and analyzing network traffic between the toys and their servers, examining the security of Wi-Fi setup processes, and performing static analysis of companion apps to identify requested permissions and embedded trackers.

The team also reviewed privacy policies and terms of service and sent subject access requests to toy vendors to test compliance with data protection regulations. This multi-faceted approach provided a thorough understanding of each toy’s security, privacy, and transparency features.

Key Results

The study revealed significant concerns across multiple areas. In terms of security, while most toys used encryption for internet communications, local network connections were often unprotected. Some toys had vulnerabilities in their Wi-Fi setup processes that could allow attackers to intercept sensitive information. Privacy issues were prevalent, with many toys collecting extensive analytics data and unique identifiers, enabling detailed behavioral profiling of children.

Companion apps often request unnecessary and sensitive permissions. Transparency was lacking, with privacy policies frequently vague or difficult to access. Only 43% of vendors responded adequately to subject access requests within the required timeframe. Most toys fell short of full compliance with the GDPR and the upcoming Cyber Resilience Act, particularly in areas of data minimization and user rights.

Study Limitations

The study has several notable limitations. The sample size of 12 toys may not be fully representative of the entire smart toy market. The research was conducted at a single point in time, preventing the tracking of changes in data collection practices over time. The team did not attempt to extract or reverse engineer device firmware, which could have revealed additional insights.

Additionally, the study’s focus on toys available in the European market means the findings may not be fully applicable to other regions with different regulations and practices.

Discussion & Takeaways

The researchers emphasize the critical need for toy makers to prioritize privacy and security, adhering to best practices in security and privacy engineering. They advocate for easier subject access request processes and more granular consent options for users. The study suggests implementing standardized privacy and security labels for toy packaging, similar to nutrition labels on food products.

For parents, the key takeaway is the need for increased vigilance when purchasing and using smart toys, including carefully reading privacy policies and being cautious about granting app permissions. The research community and regulators are called upon to provide more support for toy makers, potentially through the development of standardized guidelines or certification processes for smart toys.

Funding and Disclosures

The research does not mention specific funding sources, and the authors declare no competing interests relevant to the study’s content. The research was conducted by academic researchers at the University of Basel, suggesting it was likely funded through standard university research channels rather than by industry or specific interest groups.

About StudyFinds Staff

StudyFinds sets out to find new research that speaks to mass audiences — without all the scientific jargon. The stories we publish are digestible, summarized versions of research that are intended to inform the reader as well as stir civil, educated debate. StudyFinds Staff articles are AI assisted, but always thoroughly reviewed and edited by a Study Finds staff member. Read our AI Policy for more information.

Our Editorial Process

StudyFinds publishes digestible, agenda-free, transparent research summaries that are intended to inform the reader as well as stir civil, educated debate. We do not agree nor disagree with any of the studies we post, rather, we encourage our readers to debate the veracity of the findings themselves. All articles published on StudyFinds are vetted by our editors prior to publication and include links back to the source or corresponding journal article, if possible.

Our Editorial Team

Editor-in-Chief

Chris Melore

Sophia Naughton

Associate Editor

Related Content

Teflon flu cases surge: What you need to know

August 30, 2024

5 Best Light Beers, Ranked By Reviewers

60-year mystery solved! NASA scientists finally measure Earth’s long-hypothesized electric field

Leave a reply cancel reply.

©2024 Study Finds. All rights reserved. Privacy Policy • Disclosure Policy • Do Not Sell My Personal Information