Skip to content

Of statistics class and judo class: Beyond the paradigm of sequential education

In judo class they kinda do the same thing every time: you warm up and then work on different moves. Different moves in different classes, and there are different levels, but within any level the classes don’t really have a sequence. You just start where you start, practice over and over, and gradually improve. Different students in the class are at different levels, both when it comes to specific judo expertise and also general strength, endurance, and flexibility, so it wouldn’t make sense to set up the class sequentially. Even during the semester, some people show up at the dojo once a week, others twice or three times a week.

Now compare that with how we organize our statistics classes. Each course has a sequence, everyone starts on week 1 and ends on week 15 (or less for a short course), and the assumption is that everyone starts at the same level, has the same abilities, will put in the same level of effort, and can learn at the same pace. We know this isn’t so, and we try our best to adapt to the levels of all the students in the class, but the baseline is uniformity.

The main way in which we adapt to different levels of students is to offer many different courses, so each student can jump in at his or her own level. But this still doesn’t account for different abilities, different amounts of time spent during the semester, and so forth.

Also, and relatedly, I don’t think the sequential model works so well within a course, even setting aside the differences between students. There is typically only a weak ordering of the different topics within a statistics course, and to really learn the material you have to keep going back and practicing what you’ve learned.

The sequential model works well in a textbook—it’s good to be able to find what you need, and see how it relates to the other material you’ll be learning. But in a course, I’m thinking we’d be better off moving toward the judo model, in which we have a bunch of classes with regular hours and students can drop in and practice, setting their schedules as they see fit. We could then assess progress using standardized tests instead of course grades.

P.S. I’m not an expert on judo, so please take the above description as approximate. This post is really about statistics teaching, not judo.


  1. I feel this way about academic writing instruction.

  2. I couldn’t agree with Andrew more. But I think that weak ordering is intrinsic to elementary statistics b/c we yet haven’t settled statistical theories and methods. By virtue of the extent of controversies in statistics communities, review of elementary statistics becomes even more puzzling. Put another way, exposure to controversies yields even more confusion. Particularly about sampling in my experience. If statisticians err in their own research, goodness knows what we non-statisticians are capable of. Thus I am even more circumspect.

  3. Allan C says:

    This seems really cool. But wouldn’t it mean having to have many more instructor hours since the same material would be taught several times in the same semester? That seems too costly.

    My favorite classes were always the ones where we were expected to learn / read the textbook out of class and use the classroom time for asking questions / getting some discussion going about the material. But then again that’s my learning style and I’m sure others who are auditory learners prefer the classroom regurgitation of the text (and for them I suspect Andrew’s Judo style would work much better). I’d be super interested to see it implemented somewhere.

    • Keith O'Rourke says:


      That is the Oxford/Cambridge undergrad tutorial system. I am aware of it as a grad student, wonder if anyone who went through it might comment here.

      • Robin Morris says:

        I was an undergrad at Cambridge many years ago now. In arts subject, the tutorial system is a little like what’s described here — lectures are more or less optional; at a weekly tutorial meeting, you read your current essay, discuss it, and then your essay topic for the next week is assigned. The level could be tailored to the student. In sciences/engineering, tutorials are more of a way to get help with homework problems (set by the lecturer) that you didn’t understand. While the tutorials were in very small groups (2-3 students), the tutorials were still based on the lectures, and it’s still a sequential learning process.

        David Mackay had a slightly different take on the Judo approach – “Everyone Should Get an A”,

    • Damon Verial says:

      It seems like the classes would have no beginning or end. This idea throws a wrench into the whole idea of meeting course requirements for graduating. Surely, Andrew isn’t implying that course evaluations should be entirely done away with in favor of simple testing? What is to stop a person from registering in the uni just to take the tests and get his degree without attending courses?

      Universities are businesses before anything, and the current system is highly profitable. Even if Andrew’s method is more effective/efficient, that very fact could make universities less profitable, especially if, as Allan mentions, teaching professors require more payment.

      • Schick says:

        Well, we know student evals are worse than useless for predicting teacher quality (Uttl 2017). So getting rid of them would probably be a good thing…

        However, your point about what we would do with students is a good one. Cognitive scientists more or less know, over 50 years of testing and meta analyses, that a hybrid strategy of programmatic learning with practice and testing of recently covered material and randomly selected review is best for learning outcomes (Paschler 2009, among a good 50 years of others). The testing incents learning the new material as well as practicing old material, the review sessions make sure it is practiced (much like an instrument or a sport), the programmatic sessions make sure you cover the material (certainly, undergrads could go away and read the book themselves, but that is asking a lot of undergrads- I wager if you just treated class as practice and testing, 90% of undergrads would fail out, and then you would get in trouble with all sorts of people).

        The problem is that programmatic teaching is cheap in terms of institutional and researcher resources compared to the costs of supervised practice, testing, and feedback. My undergraduate had required statistics tutorials. I would resist having TA’s or RA’s having to perform the same function for my undergrads as a waste of their (and my) time. It’s much easier to leave the practice portion up to the students without supervision- and I think having tutorial/TA/lab sessions should probably be more emphasized than they are. That sort of thing costs a lot.

      • What is to stop a person from registering in the uni just to take the tests and get his degree without attending courses?

        What would be wrong with that? It’s win-win if they still pay regular tuition. Students would get the certification they want in fewer than four years and the univesity gets their suitcase full of cash. Isn’t the whole point of a university degree that it’s a certification and also a source of income for the university? (I’m seriously only half joking here.)

        If you just want to learn, there are lots of options other than university degrees. For instance, I’m learning stats by apprenticing myself to the Stan project, not by taking classes. I learned computer science at university. No I didn’t, I just got degrees. Most of my CS professors in Michigan State’s comp sci department in the early 80s were so clueless they just read the books out loud and gave multiple choice exams full of wrongly marked answers (the exceptions were digital design, a graduate class in AI, and a graduate class in algebraic automata theory, and a math class in algorithms, if any of my old profs are reading this!). I didn’t really do CS classes in my joint cog sci/comp sci Ph.D. program. I actually learned how to program when I jumped into real industry at a startup (from Bell Labs, which was even more academic than Carnegie Mellon, where I was a professor). The two best programmers I know never finished their undergrad degrees.

        • Shravan says:

          “Most of my CS professors in Michigan State’s comp sci department in the early 80s were so clueless they just read the books out loud and gave multiple choice exams full of wrongly marked answers ”

          I was so happy to read this! So it wasn’t just my miserable education in India.

          Growing up and studying in universities in Delhi, I had several profs (philosophy, teaching Marxist-Leninist thinking in a left-wing university, graduate linguistics in a major Delhi university) where the “teaching” consisted of the teacher reading out his 20 year old graduate-student day lecture notes from a notebook (handwritten naturally) and us sitting in class taking dictation. I did suggest to one of the professors, why don’t you just give us photocopies of your notes? He was appalled. How would we learn?

          It feels good to know I was sitting at the top table all along and never knew it.

  4. Keith O'Rourke says:

    Well judo is a refinement of martial arts where it can be practiced with a reasonable expectation of not being maimed or killed. In martial arts training for real, everyone’s life depended on the skills of the trainees – so a real incentive to get everyone more skilled ASAP.

    And as Sameera points out – no one has widely convinced others that their methods are adequate to deal with the onslaught of uncertainty in all its awful brutality. On the other hand, the instructors’ welfare has essentially nothing to do with whether students learn worthwhile thinks (as long as they don’t complain on the student evaluations).

    (One typo purposely left in).

    • What would improve an elementary statistics course? That is the question I have been contemplating after having noted, for a year or so, that characterizations of p-values were under siege in journal articles. Therefore I think that answers to that initial question could well result through Open Science principles and practices, utilizing preprints, preregistrations, blogs like Andrew’s and Daniel Lakens’, etc.

      Obviously good writing skills are entailed in all such efforts. Just trying to distinguish the uses of terminology from one context to another is a painstaking process, requiring considerable time and effort. Steven Goodman & John Ioannidis have spoken to the need of standardizing the lexicon. That too is a priority.

      I actually see merit in requiring a course in basic logic BEFORE taking statistics. It takes a long while for logic to sink in. I’m not sure how helpful it is to assign a basic logic book in a statistics course. Rather the logic that would be helpful may need further adaptation to statistics

      • Here is the article to which I was referring in an earlier post:

        What Does Research Reproducibility Mean?

        Steven Goodman, Daniele Fanelli, and John Ioannidis


        The language and conceptual framework of “research reproducibility” are nonstandard and unsettled across the sciences. In this Perspective, we review an array of explicit and implicit definitions of reproducibility and related terminology, and discuss how to avoid potential misunderstandings when these terms are used as a surrogate for “truth.”

    • Damon Verial says:

      >(One typo purposely left in).


      Why leave this in? Is it not easier to fix the typo than to point out that your post has a typo? I don’t see a pun here.

      • Keith O'Rourke says:

        What is the most important thinG one wants to impart to students – thinK.

        • Anonymous says:

          “What is the most important thinG one wants to impart to students – thinK.”

          This comment, and the content of this blog post reminded me of the saying “the student has become the master” (if that is the correct phrasing).

          I always liked this saying, and have thought about it in relation to professors in science and their students. I sometimes have the idea that current professors may not really select for the “best” students, but perhaps more for the ones who will do what they want to do.

          I wondered if this may be related to the way PhD students are sometimes hired (if i am not mistaken): as a result of a grant the professor got for coming up with certain studies/topics/whatever. I have never understood this. For me, the PhD students should write proposals themselves, which would then result in hiring and the PhD student actually doing his/her own work and not be some sort of research assistant who executes what the professor thinks is best.

          Anyway, “the student has become the master”….I think if i was a professor, i would try and get the smartest/most talented/best students to do “their thing”, and not execute my ideas. I would teach them all i know and think is useful, but let them make their own decisions.

          Let them try to become a master.
          Let them try and find flaws in, and improve, my own work for that matter.
          Let them try and do something more useful and better than i have done, and could ever do.
          Perhaps only then, i would have fulfilled my role, and could retire with a smile on my face.
          Perhaps the goal of a master is to become a student again.

  5. Mike says:

    Having trained under this method (in Brazilian Jiu Jitsu, not judo), I found it to be extraordinarily ineffective.

    There’s no systematic introduction, so beginners can easily go for a very long time without learning the most basic and crucial techniques, or BJJ’s basic combat philosophy, or even the names of positions. The teaching portion of class is usually wasted for most students, because it’s inevitably about a technique that is at an appropriate level (neither too advanced or too elementary) for only a handful of them. Even if it’s at the right level for you, it’s still really hard to learn, because there’s no easy way to review or get extra practice. If you don’t get a good handle on the technique that day (and I never, ever did), it’s up to you to find someone willing to practice it with you outside of class, which is sometimes easy and sometimes impossible.

    The nature of the large majority of judo and BJJ students is that they are not full time students, they’re not going to be willing and able to stick to a fixed weekly schedule, they’re going to have random periods with little or no training due to other obligations, and they’re training as a hobby without a big commitment. Also basically zero BJJ or judo schools have enough students to offer 10 different semester-long classes for students of different abilities. I consider the “judo model” a massive concession to this reality and *not* a planned, pedagogically sound system.

    • Adede says:

      +100. I have done martial arts both occasionally and fairly intensely (consistent 3+ times per week). Overall, I did not consider the teaching even remotely efficient or effective.

      • mpledger says:

        Part of physical training is not for the brain but the body – it’s to create the “muscle memory” that makes certain actions doable in the right form without consciously engaging the brain to do so. It’s so that your body goes into rote memory mode when called upon so that that your brain can be left for other analysis.

        So in martial arts (or any sport) there is a lot of time spent on basic skills and movements to keep up that “muscle memory”. A lot less on learning new skills.

        Isn’t the whole point of lectures that you get the material and then spend as much time as you personally need to digest it. So someone well prepared for the course may never do anything other than attend lectures but someone who isn’t well prepared may spend hours reviewing notes, reading the text book, practicing the questions, seeing the lecturers for help, attend the study centre for help, ask a senior student for tutoring etc.

        That leaves the better prepared students time to take more courses.

        I think the students wouldn’t have the self-awareness to know what they know and don’t know to know what classes they need to go to. Those with lots of bravado would think they know everything, those with little confidence would go to every class (even if they are learning nothing and wasting their time).

        What would be the difference between a grade and a standardised test?

    • Martha (Smith) says:

      I haven’t had experience with judo specifically, but did study Aikido (another marital art) for a few years, and have a good friend who is a (now retired, except for annual talks at a math camp) math professor and has taught Aikido (including at annual U.S. Aikido camp) for about as long as math.

      One thing that seems to be common in both his areas of teaching is continual rethinking of how best to approach something, so that over the years, teaching becomes more and more effective. (Especially in Aikido, one result of this is that some students say that what he does looks like magic, but he tells them that it is not magic, but an appreciation of subtlety). I have tried to do the same in teaching math and stats.

      But (as has been mentioned in some comments) there are definitely differences in martial arts and math teaching (and based on my experience teaching both math and statistics, statistics is somewhat in between, but a little closer to math than to martial arts).

      Speaking a little from my own experience teaching both math and stats, there is definitely room for some teaching that is not as sequential as typical teaching in both those fields. I think both fields could use more non-sequential teaching, but still can’t get away with being totally non-sequential. I have been fortunate to have had opportunities to do some of this not-entirely-sequential teaching. Here are some examples:

      (In the first three, students were expected to present their solutions to the class, and respond to questions and critiques from their classmates.)

      1. I’ve taught undergraduate “problem solving” courses both for a liberal arts program and for future math teachers. These did assume a basic high school math background, but the main focus was on doing “non-routine” problems — and focusing on some of the “habits of mind” that could help build facility with these.

      2. I’ve taught a summer statistics and probability course for in-service secondary math teachers that had as prerequisite basic high school math, calculus, at least one college level “proof” course, and a (not necessarily very good) introductory stat course. I was able to arrange it so that it started out with some topics in the morning, a break for lunch and for them to work on problems, then a second thread of topics in the afternoon; after a while, the two threads merged and we ended up doing a little Bayesian statistics.

      3. I taught a geometry class for pre-service secondary math teachers, where Tuesday classes were in a computer lab using geometry software to help form conjectures. Assignments for Thursday class were to prove the conjectures. Most of Thursday’s class consisted of students presenting their proofs, with critiques by the rest of the class (and sometimes by me as well).

      4. Whenever I teach statistics courses beyond intro, I start with a review of basic hypothesis testing, either by having students read a handout or in lecture form – then pose questions such as “Does this say what a p-value is?” with options such as, “Gets it”, “Partly gets it”, “Doesn’t get it”. I tell students to discuss their responses with another student, then I take a “show of hands”, then ask a student to explain their answer, then ask for a show of hands on whether other students agree (and repeat with another student if needed).

      • From my observation, the fundamental challenge is in offering a very good introductory statistics book and supplementary materials to begin with. Pedagogy. I missed the recent ASA forum on statistics and math teaching in high schools. I believe there is a video of the proceedings available. I plan to review it.

        But the other challenge is in teaching/communication skills. The late Serge Lang, Yale & Univ. of California, Berkeley, was known to be an outstanding teacher, often nominated for excellence in teaching, based on student evaluations. He was also an unusually good critical thinker. So it’s a small set of statisticians/mathematicians that qualify in that regard. And it doesn’t seem that it’s the Ivy League universities have a monopoly on good teaching or thinking.

        I think we have to address the standardization of terminology as Steven Goodman and John Ioannidis posit. Low hanging fruit.

  6. Yuling says:

    The challenge of fitting a mixture model comes from the fact the cost (either computation or economic, depends on the context) grows exponentially.

  7. Rahul says:

    Isn’t separation of the training from testing one of the fundamental problems?

    • Martha (Smith) says:

      Good point. In (good) martial arts teaching, there is the opportunity for frequent, real time, individualized feedback about what the student did wrong/could do better. This is rare in stat courses. (But bear in mind that stat classes are usually much larger than martial arts classes. Related: Martial arts instructors usually either have a “day job” or get along on a low income.)

  8. I think the way to do this would be with an inverted classroom. You record the lectures, assign a book, then students come in for hands-on help. They can help each other and get help from the instructor.

    I’ve been thinking this would be the future for a decade now. Michael Collins tried it in CS here and told me he thought it worked very well.

    • There is no one way to learn any subject. In fact, I learned more about decision-making simply by listening to people talk informally; at lunches and in breaks at conferences. I accompanied my father to many conferences when I was a child. Particularly in summers and occasionally on winter breaks

      I got to watch how academics behave with each other; what they said to each other; how they behaved toward me; and I sometimes gave my opinion. I was raised by my father’s colleagues more so than I was by my immediate family b/c I moved into a college dorm at age 14.

      I learned that many viewpoints, seemingly well researched, were result of the sociology of the academics brought together. I think too that I saw a good deal of rivalries among academics. And I found myself intervening in academic politics b/c my father was too timid, in my view. It was not healthy probably; but I learned a lot about the sociology of knowledge in the process.

      • I wasn’t trying to imply there is one true way to teach anything. I was trying to address Andrew’s immediate question of how one might run a statistics class asynchronously. Ever since the advent of MOOCs, I have been expecting something like an inverted classroom to become more popular, if not as popular as the usual lecture forms. I have exactly one data point from someone I trust working in a related field.

        Most students never raise their hand or ask a question in even a small class, unless forced to do so. Andrew does a lot of this small group work asking students to interact with the rest of the class regularly, which seems to work very well, presumably because it makes the learning more active—the students have to engage with the material, each other, and with Andrew.

        If you’re just going to lecture and only take questions from a few people, it seems both biased (towards those brave and/or extroverted enough to ask questions in class) and wasteful if you do it regularly (as in the eight straight years I taught one of our qual classes at Carnegie Mellon, or the tens of thousands of professors preparing intro calc classes). I believe my lectures got better for a couple years, peaked, then I got bored. I did wind up writing a textbook, though, which was nice, because now others can learn the material on their own. I think it would’ve been even better had a left a Coursera-style self-study course with recorded lectures.

    • Ethan Bolker says:

      This is called a flipped classroom: (and many other links).

    • Greg Francis says:

      I tried something like an inverted classroom a couple of years ago for Introduction to Statistics in Psychology. The class period was to answer questions and help solve problems. When there were no questions I had some material prepared on various topics or demonstrations.

      The course was a disaster. Many students were so confused that they could not even ask good questions. Nearly every class period ended up being me going through homework problems; often with the student discovering that she had made some trivial error.

      By the end of the semester the students were begging for lectures.

      Maybe this kind of inverted classroom can work well, but I think it is not trivial to set up.

      • Martha (Smith) says:

        I agree: it is not trivial to set up — nor to implement. But there are various ways of facilitating student involvement.

        In one class I taught, the standard textbook was poorly written. Since I wanted to give reading assignments followed by class discussion, I made up “study guides” for the assignments — in particular, pointing out places where the students had to read particularly carefully; posing questions for them to think about as they read, etc. It worked reasonably well — but I also had to do things like “On Wednesday, these students will be first in line to be called on,” and after a student attempted to answer a question, ask the class, “Do you agree or disagree? Why or why not?” or “Can you add anything else?” Most students (especially at first) don’t magically engage themselves without some guidance. They often don’t know what questions to ask, or don’t think they are competent to ask good questions, or are afraid of being put down. So things like, “Can anyone help X out?” or “Does anyone have anything to add?’ or “Can anyone explain it another way?” are important.

        Another type of teacher “leading” is to ask questions with a show of hands — e.g., “How many people agree with this student’s explanation,” then call on someone who does or does not agree to explain their answer. The idea is to get them involved collectively, not just individually. Students can often learn from each other, if they are given some assistance in doing so.

        Another thing that facilitates good classroom participation is to wait long enough after posing a question in class to give students enough time to figure out how to say what they need to say, or to get up the courage to say it. I once was told to count silently to 5 after asking a question before calling on a student. That was poor advice — it’s better to wait till a few students have raised their hands, or to count silently to 30 or 60 before calling on someone.

        And also be sure to thank a student if they point out something you have done wrong.

        • Shravan says:

          This is really good advice, Martha, I will follow it.

          I had a math prof in Ohio State who always gave a student a 1/2 point on the final grade for every mistake we caught on her slides. This led to people paying extremely close attention in class, it was almost a game to find the prof’s mistakes.

          • Shravan says:

            After the course was over, I had a strong suspicion she played us. I think she put in the mistakes on purpose.

            The Stan guys did this too in Paris once (not sure if it was intentional). They gave out printed example code, where the parameters block had alpha as a parameter declaration, but the model block had a. LIKE AN IDIOT I fell for it and copied out the wrong model and of course it didn’t compile. It’s a great trick that I will try in my next class, although I might annoy some students.

            • Interesting idea in terms of adding mistakes to get people ot pay attention. But I have a hard time with grading-driven education, which is probably why I failed miserably in my first and only undergrad teaching—I’d inherited a first-year psycholinguistics course (not my idea!) from Ted Gibson when he moved to MIT (my whole department collapsed the following year and we all quit, ending my tenure as an actual professor—industry was too tempting during the first dot-com boom because I could put my natural language processing skills to actual use).

              And yes, those mistakes in the program are intentional. I can’t remember if that was Andrew’s, Daniel Lee’s, or Michael Betancourt’s idea originally, but we tend to do it a lot in our hands-on material to get people to think. Not to give too much away, but we also tend to do that with the data/model fits as well as the code. I’d say it’s more like we exploited expectations than that you were an idiot. We wanted people to be surprised and have to wrestle with things.

              It’s too easy just to present things that work. That’s why I really like Andrew’s Stan lecture on his world-cup model, because he takes you through is own debugging process of fixing a couple problems in the first formulation of the model. This is really hard to do in a lecture course, but Andrew did show it was possible.

              We’re revamping all the Stan docs. Dan Simpson and I are starting a general overview of recommended Stan methodology (I hate the term “best practices”, but that’s essentially what it is). Andrew and I are rewritng the user’s guide to make it more micro-case-study oriented rather than a bunch of disjointed tips. Dan and I are trying to figure out how to add things that are wrong and how you can use diagnostics to find the problem and then statistical techniques or programming techniques to fix the problem. Michael Betancourt’s courses are really fantastic at this in my experience. I can never get over how he manages to pack so much useful advice into a day by mostly leading people there themselves through practical exercises.

              There’s a really tough decision between drills and new material, too. To use the analogy of my favorite role-playing blog, the Angry GM, if you want to teach someone to play hockey, you don’t put them in skates in a game and ask them make slap shots. You start by teaching them to skate in a straight line. Then you work on turning. Only then do you get a stick, but you don’t get a puck yet. You have to learn to skate with a stick (actually pretty easy, especially if you grew up in Detroit when the yards froze over). Only then do you even think about a puck. But you don’t get put into a game. No, you first have to learn how to skate and shoot. Daniel Lee’s our drill master, because he’s actually a basketball coach in computational statistician’s clothing.

              Mitzi’s grad school in linguistics at Santa Cruz tried to lead the students along under their own power. She still jokes that depending on who taught the course, the students magically recreated standard transformational grammar or relational grammar. I think that means if you can see the strings too clearly, it won’t work so well.

              • Martha (smith) says:

                “You have to learn to skate with a stick (actually pretty easy, especially if you grew up in Detroit when the yards froze over)”

                Ah, brings back memories of growing up in Detroit. Virtually all the kids had skates (even if hand-me-downs) and the boys also had hockey sticks and pucks. A neighbor who was a physician and had three sons had a large side yard which he flooded every winter. The boys in the neighborhood spent a lot of time there using their skates, sticks, and pucks.

              • AllanC says:

                To nitpick as a Canadian…the proper lingo is for one to “take (not make)” a slap shot!

                But Bob’s status just went up in my book now I know he is a fan of the game. +1

              • AllanC: doesn’t “take” refer to the action of attempting, and “make” refers to success? Or is that not the hockey lingo?

              • AllanC says:


                If you didn’t complete one of the sequences of actions that fall under the banner of slap shot you didn’t really take one. You endeavored to do so but fell short. In such circumstances you usually just served up a muffin that led to an odd man rush. An alternative explanation would be that you tried to take a clapper but chipped nothing but ice.

              • Sure, that makes sense. But if you do successfully take a slap shot (on goal) but you don’t make the goal, then you didn’t make a goal with your slap-shot. I would think you might hear someone describe this as “he took a slap shot, but didn’t make it”

                but I could be wrong. I certainly would hear that kind of thing in soccer (my preferred sport), such as “he took a shot from the top of the box but didn’t make it”, now if he tried to take a shot from the top of the box but he missed the ball… he wouldn’t have even “taken the shot”

              • Martha (Smith) says:

                Never argue with a Canadian on terminology concerning hockey.

              • Martha: I’m not arguing, I’m asking for clarification from the experts ;-)

        • Guido Biele says:

          I have had positive experiences with using clickers in lectures (in psychology/neuroscience) and found clickers particularly useful for two things:
          – check if the students understood the key concepts I tried to convey (if no, I would go again through that material in the same lecture)
          – keep students engaged.

          What I especially like about clickers is that they facilitate getting responses from shyer students, and that the make it harder for students to simply imitate other students responses by waiting for others to raise their hand first.

          I first read about clickers in a science article about their use in STEM teaching, described here:

          • AllanC says:

            During my first year physics course there was a clicker component. It was required (if you didn’t click on some percentage of the questions, 75% I believe) you would fail.

            We ended up giving one person 15 or so clickers and we would all rotate going to lectures. It worked because the entire first year engineering cohort had the same physics class.

            The times I did show I didn’t enjoy the clicker questions as they seemed rather trivial. In my experience it was hard to cultivate genuine engagement with the types of questions one could ask via clicker. But maybe there is a better way to frame the questions that actually stimulates.

            • That was like my intro computer science. Dave Lewis and I started alternating lectures, then alternating weeks, and finally just showing up for the final. It went that slowly. Luckily, after my first quarter, Michigan State Honors College let me opt out of any general ed or intro requirements and just jump into advanced classes, so I got to take a ton of graduate classes (micro-econ, cognitive anthropology, random graph theory, AI, Montague semantics, a seminar on pragmatism focusing on Rorty’s Philosophy and the Mirror of Nature, etc.). By their very nature graduate classes are much more engaging. We had to formulate a pop concept experiment in the cognitive anthropology class—that got everyone in the class very engaged in methodology and we all had skin in the game because we were doing real experiments.

              Sorry, Shravan, it wasn’t all bad. Mostly it was great outside computer science and the gen ed courses. Most of my math classes (like algebra, analysis, topology, set theory, logic) had only 10 or 12 students because there were only about 50 pure math students out of 30K or so undergrads. So small class size really helps, too, if you want to engage students.

              • Shravan says:

                Bob, no, it wasn’t all bad for me either. My BA Honors program in Japanese was phenomenal. When I moved to Japan after the three-year program, I was in a class with 30 or so people like me, who’d just finished a first degree in Japanese, from all over the world. I found I was very well trained, only lagging behind a guy from Prague, and another guy from the Netherlands, and a couple of French students from Paris. Also, my linguistics training in India was generally excellent, with some exceptions. For a country so poor, with a semester fee of 1 Euro, my education was just unbelievably good. I was talking about the cases that were so bad that I thought they must be unique to India, and I was thrilled to learn the US is no better.

        • Martha (Smith) said:

          I once was told to count silently to 5 after asking a question before calling on a student. That was poor advice — it’s better to wait till a few students have raised their hands, or to count silently to 30 or 60 before calling on someone.

          Waiting an uncomfortably long time lets the audience know you mean and inevitably someone winds up breaking the ice. It’s like a magic trick in my experience. Often once the ice is broken, I get a flood of questions.

          In terms of more proactively engaging the class, Andrew does a fantastic job by having people break down into groups of 2 or 3 to work through a problem then he as them present their answers. I think it strikes a nice medium between individuals having to speak up individually and raising hands. It also engages everyone more than the usual “does anyone have any questions”?

        • Jeff Valentine says:

          To add to Martha’s suggestions, I have a friend who uses the inverted model, and he strongly recommends low-stakes quizzes that are due prior to class. Even though they are low-stakes, he feels that it is enough of an incentive for students to do the needed work beforehand, and it also gives him ideas about where students need help.

      • Shravan says:

        I have done this (inverted/flipped class), the students liked the availability of video, because they could pause it and replay it. But my audio equipment was not good quality then and that proved to be a crucial problem: whenever I turned away from the computer to the blackboard, my voice would become faint (now I have high tech video streamer quality stuff, thanks to my son’s advice on how to become a youtube star and make lots of money).

        One thing I did was to review in class the contents of the lecture they had watched at home, in a few minutes. This primes them to get ready for the work ahead in class, and primes them to ask questions.

      • Rahul says:

        I suppose a big part of the problem is student motivation. It takes non-trivial self-discipline to do the assigned readings before coming to class for an inverted classroom.

        Part of traditional teaching is something of a drill-sergeant / motivational coach role.

      • Elin says:

        Managing an active learning classroom of any kind is extremely hard and requires a lot of planning and constant revision. In the end you need a tight schedule and to know exactly what questions *you want students to ask* and to set up activities that will encourage them to ask. You want to know precisely when you will bring the whole group together for a 5 minute lecture or when you will put student work up on the screen. Having taught this way for many years I know what topics we’ll need to stop and talk about as a whole group and what questions I’ll need to ask each group to think about as I walk around the room observing what they are doing.

  9. Shravan says:

    Thinking about the Judo analogy, it occurs to me that as a Judoka if you are truly crappy, you will often find yourself lying on your back staring at the ceiling, or have your face and ears burning from being dragged over the rough surface of the judo mat. The feedback on your skill level is quite immediate and unambiguous.

    In the use of statistics in science, however, you can and often will be rewarded with millions of Euros/Dollars of funding, and a high h-index, and you may never ever get any feedback on crappy your stats skills really are and just how badly the data defeated you.

    • Shravan,

      It looks like such extravagant funding for statistics [more generally social sciences] is not going to continue. Moreover it seems that since the 90’s there has been vigorous debate in some academic circles. But it is going to take even more transparency as possibly afforded by Open Science collaborations. I gather from the collaborations that John Ioannidis has been forging that even more rigorous discussions will ensue. Even the public has more access to prominent expertise. And as Nassim Taleb and the late John Rawls encouraged was to include more creative eclectic thinkers in the mix. Hobbyists.

  10. Dan F. says:

    Maybe a better example to think about is instruction in musical performance.

    In such a context it is not normal to have beginners practicing with experts.

    Even though professionals begin practice by warming up their warmups require technical skill that students are far from achieving.

    A big difference between learning a scientific discipline and learning judo is precisely that, because of the plainly hierarchical, sequential nature of learning a scientific discipline, such discplines are greatly more developed and pursued far more deeply than something like judo. The judo approach never leads to building deepwater oil drilling platforms.

    • Andrew says:


      Sure, but my point is that learning statistics is not hierarchical or sequential. Actually in statistics we keep learning the same messages over and over again, at different levels. We teach to Ph.D. students many of the same lessons that we teach to high school students.

      • Teaching of statistics is neither hierarchical nor sequential, as Andrew notes. In hindsight, it’s the dirth of explanations of theory that is a drawback. Rex Kline attempts in a few chapters in his textbook, Beyond Significance Testing.

        I venture. Rather a scaffolding approach may work better whereby some combination of thinking curricula enables students to understand assumptions attendant to principles, in the very earliest stages of learning statistics.

        That is at least how I learn best. I am unsettled unless I have grasp of theory/theories inherent to a particular rule or practice. Takes effort.

    • Music’s a good analogy because it’s a similar combination of theory and practical skills. I think one of the hard parts of stats education in general is that education in anything other than theory is relatively rare. For instance, Columbia stats brings in The Carpentries (formerly Software Carpentries and Data Carpentries, and whatever) to teach their incoming Ph.D. class hands-on skills for a week or so. It’d be like if you were going to be a professional musician and you got a week of hands-on training before you start about how to assemble your clarinet and honk out a few notes and put your fingers on it to make a C sharp, then you got a ton of theory lectures, and then had to go and become a musician. Not easy.

      Mitzi’s dad played French Horn for the Cleveland Orchestra for 30+ years. It was always almost alarming listening to him warm up. It’d go from a few tentative notes, to scales, and then all of a sudden he’s blasting solos from Mozart’s horn concertos. I heard his students sometimes, too, but they were also very advanced.

      At least music sometimes leads to orchestrated symphonies, if not oil rigs or space ships.

  11. Keith O'Rourke says:

    I was thinking of traditional martial arts circa 1800s when I wrote this ” In martial arts training for real, everyone’s life depended on the skills of the trainees – so a real incentive to get everyone more skilled ASAP.” The variation in today’s for profit or ego run martial arts clubs vary greatly in quality and teaching. As well as how traditional they are and it what sense. For instance, my first club largely was financed by an endowment

    I believe Shravan’s comment about the feedback on/for learning is very important – its hard to incentivize most students beyond getting what ever grade they have targeted. And the ones that really, really want to learn seldom will get feedback directly from the reality of doing research. (Also, in kickboxing its hard to convince oneself that you could have ducked a punch/kick but you just couldn’t be bothered! There are martial arts other than Judo or Kickboxing where you can’t really spar without risk of injury and that level of feedback is missing.)

    Now Zad Cow’s link was interesting but almost any world champ has within their training group a brilliant trainer/technician and strategist. As one world champion once mentioned “thinking back I lost that fight before the first round. Embarrassing by I think true.”

  12. Andre says:

    From my experience, in boxing, you can still go in the ring with guys with more experience. They generally will not “hurt you,” because their skills are at a level where there is no question as to who would “win” the fight. They “work with you”. Likewise in BJJ, the more experienced guys help you out, teach you things along the way, they don’t expect you to know everything.

    Academia, my experience has been the opposite, more similar to Bobs. Professors write the book on the board, humiliate you if you seem like an easy target who doesn’t know their shit. So I stopped going to class all together. No need to waste my time (and $) attending class.

    Anyway, frustration from classes is part of why I started getting into fighting. I noticed boxers and BJJ practitioners are A LOT nicer than academics. They ask me how my day was, make sure I’ve got a community to look to. Funny how that works right? More respect for one another in that world, I think.

  13. Karina Brotto Rebuli says:

    I’ve already wondered about this! I teach in formal education and also in a method for self development, similar to martial arts.

    The professor’s ability on teaching in a non-sequential course should be improved as well! This maybe will be the main barrier, because not only the student will have to be more engaged, but also the professor – really committed with students development. Prepared, so, for teach formal stuff “on demand”, according to the day/group. Sometimes in a more guided lecture, sometimes in a lecture that will focus on students needs.

    The levels division works very well to this kind of education!

    And, last but not least, this kind of course motivates more the professor to improve itself also! :)

  14. gregor says:

    I don’t know about judo, but I will offer my critique of basic statistical education.

    Below are the stats courses I took as an undergrad.

    -a very basic intro stats course (similar to AP stat syllabus)
    -a condensed one semester probability and statistics course (with calculus)
    -an econometrics class (regression, had matrix algebra)
    -a one-year probability and statistical inference sequence

    The problem with the intro class is that they barely cover any probability and jump right into samples and statistical tests that the students can’t possibly understand. This means it ends up being rote application of statistical tests. The next course was somewhat better but it was just too much material to fit into a semester. The econometrics course was decent, but I think it would have been better if I’d had a better grounding in statistical inference.

    Finally, I took the one-year probability and statistics sequence and I realized that I did everything backwards. Having a good grounding in probability (at say the level of the Ross book) makes all the rest of it make a lot more sense, imo. If I did it all over, I would have taken the probability course first and only then bothered with any other stats courses.

    • Martha (Smith) says:

      I agree that a good grounding in probability is needed to understand statistical inference. The way I summarize it:

      1. If something involves statistical inference, then it involves uncertainty;

      2. Probability is a way of talking about uncertainty in coherent quantitative terms.

      • Sameera Daniels says:

        Martha, then do you may be in disagreement with Valentine Amrhein

        • Sameera Daniels says:

          Sorry Martha! I hit the submit too soon. I’ll have switch to laptop to reference the article I was referring to. Be back

        • Martha,

          I meant this article. Specifically should we assume that ‘probability’ is coherent, given the argument laid out in the following article?

          Inferential statistics are descriptive statistics
          Valentin Amrhein, David Trafinow, Sander Greenland

          There has been much discussion of a “replication crisis” related to statistical inference, which has largely been attributed to overemphasis on and abuse of hypothesis testing. Much of the abuse stems from failure to recognize that statistical tests not only test hypotheses, but countless assumptions and the entire environment in which research takes place. Honestly reported results must vary from replication to replication because of varying assumption violations and random variation; excessive agreement itself would suggest deeper problems, such as failure to publish results in conflict with group expectations or desires. Considerable non-replication is thus to be expected even with the best reporting practices, and generalizations from single studies are rarely if ever warranted. Because of all the uncertain and unknown assumptions that underpin statistical inferences, we should treat inferential statistics as highly unstable local descriptions of relations between assumptions and data, rather than as generalizable inferences about hypotheses or models. And that means we should treat statistical results as being much more incomplete and uncertain than is currently the norm. Rather than focusing our study reports on uncertain conclusions, we should thus focus on describing accurately how the study was conducted, what data resulted, what analysis methods were used and why, and what problems occurred.

          Amrhein V, Trafimow D, Greenland S. (2018) Inferential statistics are descriptive statistics. PeerJ Preprints 6:e26857v2

  15. renato says:

    One of the main points that your model left out is that in martial arts classes there is not only one instructor.
    There is a hierarchical structure based on skills (sometimes marked by the belt which the student wear), and anyone that is more graduated than you usually can help you by pointing your mistakes or giving some tips.
    This reduces the teacher workload, and provides much more relevant feedback to each student.

    In a academic environment, this would require to have PhD, Masters, undergrad and high-school students in the same class, or at least students from several different years.
    I guess this is easily implemented for longer courses that spans more than a semester or year (like language courses).
    However, it looks harder for courses that are supposed to teach some extra skills, but are not the main subject in a major.

    • Andrew says:


      Good point. The typical college class has just one instructor, or two if there is a teaching assistant. I agree that it would be better to structure things so that the students in the class can be both teachers and learners.

  16. Sameera Daniels says:

    I like Renato’s class composition. I would qualify as a weely weely class dummy. Lol

Leave a Reply