We just started using Crazyegg at the Khan Academy, and I made this video to share with other folks at KA to introduce them to the tool and summarize the interesting stuff from the baseline snapshots. It may or may not be extremely boring for everyone else, but I figured I would post it since there are some kinda neat insights into how the site is being used in there.

Let me know in the comments if this was helpful/interesting.

Mobile. Social. Local. Pivot. - Well one of those, anyway

I haven’t really written about the somewhat massive redesign that the Khan Academy site underwent in January/February, but in case you didn’t notice, it changed:

A lot:

The redesign accomplished a ton of goals that the organization had, and also gave us a chance to make typography, layout, and navigation a lot more consistent visually and technically. It’s a bunch easier to add/change content which is good, because we’re doing a lot of that. And people didn’t hate it either, which is every designers fear when you’re working on a product that people love.

But that’s not really why I’m here today. I’m here today to talk about “the future.” More accurately, “the present,” but saying, “the future” always seems so much more romantic. 

Introducing: the Khan Academy mobile site

One thing that the new design didn’t address was the mobile experience, especially on smaller screens (it works reasonably well on the iPad). As of this moment, if you hit the site from a device running a mobile OS like Android, iOS, or WebOS (sorry Windows Phone 7 folks, we’re working on it), you should be greeted with a new version of the site that is, for the moment, focused on one thing browsing/watching videos:

A couple of things worth noting: this isn’t a separate site. It’s the same site with some mobile magic applied to it (I will reveal those secrets in just a moment). Features that are missing (login, discussion, exercises) were intentionally cut for v1 since we were really looking to validate that this was a) possible and b) people actually liked it enough to warrant the effort of making the other features work. We did take the time to make it easy to switch between the mobile and normal sites so you can parachute out of the mobile stuff if it isn’t working well enough for you. 

How we built it

jQuery Mobile did basically all of the heavy lifting for us. The thing that made it easiest for us is that the markup for defining pages and list views is pretty unobtrusive, and with a little device detection to load the right CSS and JS, courtesy of Ben, we only had to do a tiny bit of conditional styling inline. If you inspect the homepage, you’ll see that we’ve defined a bunch of “pages”: one for primary navigation and one for each playlist. In total, it’s about 2k of additional markup that defines all of the navigation in the mobile site. 

Once you grasp how navigation and linking work, the experience of using jQuery Mobile is something akin to sprinkling magic fairy dust on our existing site and having a mobile site grow out of that. And even though we’re not doing everything by the book, it’s surprisingly solid.

If you want to see the specifics of the implementation, you can check out the code.

Still to do

Even before we add new features, we’ve got stuff to do. The performance on some of the longer lists is a bit sluggish and can cause a JS timeout in some cases. We need to figure out if that’s because we are generating the playlist “pages” from the *very* large homepage (clocks in at about 530k) or if we should be doing the markup differently or what. We’ve also got a bit of a kludge in there to help us handle linking to resources that haven’t been converted to mobile pages yet (we show the standard site for those pages). Finally, I’ve got some work to do creating a custom skin that more closely matches the branding from the standard site, but it looks like there’s a good way to do that.

So give it a shot, and let me know what you think. Feel free to leave comments here, but it would be awesome if you would report your bug or feature request in our issue tracker.

My take on common misconceptions about the Khan Academy

With all of the positive press the Khan Academy has received lately, we’ve also started attracting a bunch of new critics. This is a good thing. I can’t tell you how existing in an echo chamber where everyone loves everything you are doing can make a sane person become really paranoid after a while. While there are a bunch of really valid concerns about what we’re doing, I wanted to try to tackle some of the more pervasive misconceptions about the Khan Academy from my perspective.

Misconception 1: The Khan Academy is trying to replace teachers

This is totally false. We have no desire to replace teachers. Teachers are in a unique position to understand their students in a way that a computer program has no hope of doing any time soon (maybe post-singularity?). We firmly believe that this understanding is important from both an educational and social perspective, and if anything, we hope to improve it.

Our goal is to offer tools that will provide great teachers with better assessments and help in motivating students to take charge of their own learning while creating more time for project/concept learning and greater opportunity for truly differentiated instruction. Most great teachers are already trying to find ways to provide these things to their students. We think we can help them.

As for the bad teachers — the unqualified one, the one phoning it in, the one with no passion for the craft of educating — we’re on a mission to make sure that those teachers’ students have a way out. In other words, if the Khan Academy is a substitute for what you’re doing, you’re doing it wrong, and we’re going to make that much more obvious. 

A smart guy once said to me, “The Khan Academy probably won’t replace a truly great teacher, but it could make the world bad-teacher-proof.” Exactly.

Misconception 2: We think students should use the KA all day, just like they go to class all day

Again, totally false. There’s a shared belief around here, based on many combined years of educational experience, that many teachers have a tendency to overcomplicate things. That combined with a lack of differentiated instructions leads to an incredible amount of wasted time in traditional educational settings. We think it’s possible to spend much less on instruction if the instruction is actually targeted at what a student needs to understand right now to get her to master a concept.

The dream is to reduce the amount of instruction time overall, but especially to reduce the amount of time spent on broadcast instruction. Instead, use that time for targeted intervention and project learning/concept application with guidance of peer tutors and the teacher. 

Misconception 3: The Khan Academy is a mechanical how-to guide that doesn’t teach ideas

I’ve seen this one a bunch recently, and I’ve been trying hard to figure out exactly where it comes from since Sal makes such an effort to include concept development in his videos, and we rarely hear this complaint from students. In some cases, and I suspect that this isn’t actually that uncommon, it’s just confusion from not actually watching the videos all the way through. I think this one is driven by expectation. Teachers have seen resources that seem very much like the Khan Academy before, and there is a tendency to believe that our videos are like all other videos. In practice, this isn’t what we’re seeing. We’re constantly getting feedback from students that the videos make the difference in “getting it.” Watching only a small portion of randomly selected videos might not show you the concept development. I encourage people to dig a bit deeper before dismissing the work that’s been done on this.

None of this is to say that we can’t get better at this. I’ve seen an interesting suggestion recently about showing common misconceptions in the videos as a way to increase engagement and the chances of deeper learning more quickly. I just happen to believe, based on the feedback we’re getting from users, that we’re not missing the mark here as badly as some suggest. 

One other thing: Mechanics have their place, especially in developmental math. Some things need to be able to be done quickly and without much thought. If I need to reason out how to multiply things every time I do a multiplication problem, I am going to be some pretty deep doo doo. After seeing the number of students that struggle with multiplication facts, I am inclined to say that we shouldn’t be dismissive of this either. If we can find ways to make this practice more engaging (as we’re trying to do) maybe it will take care of itself in time.

Misconception 4: The Khan Academy is all about changing school

This one is a bit subtler than the others. While I believe that the Khan Academy can be used in the classroom to great effect, it is not actually about school, in the institutional sense. It is about learning.

Since joining the Khan Academy, I’ve spent a lot of time developing stuff that’s helpful to teachers (who typically teach in schools, but we get a ton of homeschoolers too), and I’ve done a bunch of work making things better for students. At no time during design or development did the team make a decision that would benefit schools at the cost of motivated learners visiting us out of curiosity, or homeschoolers, or all of the bankers that come just to watch videos about currency issues with China and the financial crisis. 

When students and teachers/coaches engage with us, we take their input and feedback seriously no matter where they teach or learn. Our goal is to democratize people’s ability to become educated. This doesn’t mean that as an organization we won’t work to help schools implement the Khan Academy in their classrooms, and institutional change in schools may very well be affected by the work we are doing, but we don’t kid ourselves about this: that kind of change will be the result of an incredibly lengthy process, and improving access to a high-quality education is just as important to those outside the school system as in it.

Misconception 5: We think that the system we’ve developed today is the one that is going to get us to our end goals

It seems like a lot of people look at the Khan Academy and see a finished product of some kind. I think that is due in part to Sal having developed so much content over the last several years (what a jerk, right?). From the outside, we don’t exactly look like a startup, but culturally and organizationally we are. I think if a VC were to meet us today, they would see a smart motivated team with great (charismatic) leadership that really believes that they can put a serious dent in the problem of access to high quality educational experiences for everyone. We are scrappy. We are obsessed with delivering a great experience. And we are open-minded. We are not arrogant. We do not believe that we have all the answers. I think that VC would look at us and the success we’ve managed so far — and then they would get all sad because we’re a non-profit.

Seriously though, the critiques of the .9 beta version (the version number I have just made up out of thin air to describe the state of the site) as not being the be-all end-all of educational tools remind me a lot of the critiques that people have about startups in the early stages. There’s a lot of good stuff to come. All we can do is work our asses off and hope that we continue getting people to love the site enough and use it enough to give us the time we need to execute on our plans.

In summary…

The Khan Academy is not trying to get all teachers fired. We do not want to see kids sitting in front of computer screens all day. We strive to make sure that we are teaching concepts and will continue to work at this. We’d love to see institutional change in schools but we have a much more pressing mission of making high quality education accessible to as many people as possible as quickly as possible. And we are nowhere near convinced that we’ve gotten everything exactly right today and spend every day finding ways to make it better. 

In the end, whether we succeed or fail, I am thrilled to be working on a project that is fueling the kind of discussion and debate. I am hopeful, and perhaps even overly confident, that the Khan Academy will continue to be a positive force in that debate. But what kind of startup would it be if the employees didn’t believe in the mission and the company’s ability to deliver?

UPDATE 1: Tried to clean up the phrasing on #4. It was causing some confusion about whether or not we were planning to get KA implemented in schools. We are, we are just designing with a bigger audience in mind.

Khan Academy and LASD Pilot on Gates Notes

khanacademy:

A few weeks ago, the Gates Notes team came down and interviewed the team here at the Khan Academy and all of the great students, teachers, and administrators participating in the pilot program over at the LASD. We’re excited to be able to share this with everyone because of how this experience has helped to inform many of the recent improvements to the Khan Academy. The Gates Notes team managed to capture a bunch of really interesting footage, so make sure to check out all of the videos:

http://www.thegatesnotes.com/TED/Speakers-Topics/Sal-Khan/

Khan Academy Profiles: You are what you know

About a week ago we launched Khan Academy user profiles. Profiles are meant to round out the previous work we’d done on the design of the knowledge map, exercise interface, and badges by bringing all of the information about your performance on the site into a single interface.

There were a few competing design goals here, but the most important one was to flesh out what it means to “progress” within the Khan Academy. At a high level, the page breaks down into a few major sections

  • A set of top level stats that emphasize completion of exercises and videos
  • Vital Statistics: A completely new set of tools for measuring work: Activity, Focus, and exercise completion
  • Achievements: A clearer summary and more interactive badge explorer
  • Recent Activity: Pretty straight forward, but coalesces information that was previously hard to see

The original version of Khan Academy included stars (proficient exercises) and points (a measure of effort). When we added badges, and the badge summary page, it turned into the easiest way to measure progress because that information was so neatly summarized for users. It was never our intent to have badges be a primary motivator for the majority of users, but the lack of a profile was mildly distorting the experience for some users. While badges, stars, and points remain important, we wanted to make profiles a powerful metacognitive tool for students in the same way that the knowledge map helps them understand the interconnectedness of the topics in Math.

Don’t dumb it down

Kids are smarter than we give them credit for. No one likes being patronized at any age, but all too often software patronizes students. Things like overly kid-like color schemes, oversized buttons, and over-simplified user interface are only really appropriate for *very* young children because they send a clear message to everyone about what you think of their level of intelligence/expertise. After spending several hours observing and interviewing 5th graders, I am happy to report that they are more sophisticated and willing to experiment/explore than many adult users I’ve worked with. 


The Focus graph shows how a student has divided their time on the site over the selected time period.

This is a complex graph in some ways, but it also engenders some visceral reactions, like, is it simple looking or busy? A student/teacher might not at first glance notice that on the exercise ring we show a star next to any module name that you’ve earned proficiency in, but when a user hovers the exercise the tooltip will explain what the star means. Our goal here was interesting at a glance but not necessarily completely obvious. The other important design decision on the graphs is that we didn’t use absolute scales anywhere. The result is that whether you spend 5 minutes or two hours on the site each day, the information remains visually interesting and useful.

Make it cool

We want students to start to identify themselves with the information that we’re showing in the profile. We want them to think it’s cool to have finished 80 math modules. As Ben has pointed out, grades are kind of boring in the grand scheme of things. Khan Academy profiles are designed to be visually compelling and highly interactive. Every chart provides tooltips and drill down capability (with back button functionality preserved). This gives a feeling of depth and allows students to explore/discover the connection between different metrics. 

And what’s cooler than success, right? It’s one of the most addictive feelings in the world. So many students struggle with the feeling of failure, and constant absolute measures like grades only exacerbate the problem. Not everyone is going to move at the same pace, and so we owe it to students to find meaningful and interesting progress indicators that include both absolute and relative measures. Profiles were specifically designed to provide measures that help students better understand their own educational progress, set goals for themselves, and measure themselves against those goals. The good news is that it really seems to be working.

What do you think?

In Ben’s original post on what you guys wanted to see in user profiles there were a ton of good discussions and ideas generated. Are you using profiles? If so, are they interesting to you? Missing something important? Are there things you like/dislike about the design itself (especially if you’ve had trouble/confusion while trying to use profiles)? 

To learn, quit studying and take a test

Taking a test is not just a passive mechanism for assessing how much people know, according to new research. It actually helps people learn, and it works better than a number of other studying techniques.

- Pam Belluck, The New York Times

A really interesting article, and quite in line with what we’re developing at the Khan Academy. We’re undergoing a major effort to back up our video content with engaging interactive self-paced user exercises that test you as your learn the material. We’ve already got the majority of K-12 math covered, and we’re working on new exercises every day.

In our pilot with the Los Altos School District we’re getting to test exactly how effective this methodology is with real students. Like all good engineers, we’re results driven, and the data is, thus far, very encouraging. But it can’t hurt when the research backs you up, can it? 

bjk5: Teachers have to be fearless

Ben writes:

…to dive into a new way of doing things as radical as the Khan Academy. To integrate Khan Academy usage deep into the core of existing class structure requires more than just mental agility. It requires a willingness to experiment and an excitement about being part of something new and big in…

Your challenge, if you choose to accept it...

Our exercises are seeing tons of use, but as you might have noticed they are really bite-sized chunks that closely match our video content. It’s not a one-to-one relationship, but it’s usually one exercise to just a few videos. They’re intended to provide students and coaches with timely feedback about how well a student understands an individual concept. There are no “grades” handed out, even though progress is measured, tracked, and encouraged. In other words, they are formative assessments. Rewards given for these exercises are not, generally speaking, based on the difficulty of the exercise (e.g. you can get a “Nice Streak” badge on any exercise). 

The balance to formative assessments is summative assessments. For the curious, summative assessments are designed to summarize the progress/development of a student at particular points in time. Our version of summative assessments is called “Challenges”, and we’ve designed them to represent milestones in mastering the different subject areas of the knowledge map.

Introducing: Challenges

Challenges are summaries of the most challenging exercises in each of the major subject areas of the knowledge map. To see them, you zoom all the way out on the map. 

After playing with this for a while, we landed on a very literal representation of the relationship between the Challenge and the exercises that it covers. While we’re happy with the feel of the interaction (zooming out to get to the summaries), we’re not convinced that this is the best visualization. This is something we will be watching/considering closely in the near term.

How they work

Because a challenge is supposed to be, well, challenging, and we need to assess students on a bunch of different types of questions, challenges require you to get multiple streaks of 10. The header tells you exactly how many streaks you need to get, what exercise the current problem was taken from, and shows you your progress on the new multi-streak bar. Each streak will have problems from the various exercises covered by the challenge. One important thing to note here is that we won’t give students two questions from the same exercise in a row. The added context switching increases the overall level of difficulty of the challenges. 

Understanding exactly how difficult these challenges are, we’ve built in one other feature to make the process a little more forgiving, while keeping up the same level of rigor, breaking your streak will only drop you back to your last complete streak (not to 0). That means if you get 23 questions in a row correct, and get the 24th wrong, your streak will reset to 20 instead of 0. Hints and videos work the same as they did in regular exercises with videos maintaining and hints breaking your streak.

Unique Rewards

Each challenge that we create will have a unique reward that we’re calling “Challenge Patches”. We’ve borrowed the color scheme from the “proficiency” stars on the knowledge map to create an implicit connection between obtaining these rewards achieving mastery in a particular subject area. In addition, we wanted them to feel more “real” than the other badges, so they’ve been given a very tangible appearance inspired by the mission patches that an astronaut might have sewn onto their uniform. Who knows, maybe some day we can get someone to make them into real patches or stickers?

We’re learning along with our students

Although it’s very early, it only took a few minutes of the new exercises being live for students to discover and start working on them, and even easiest challenge, the arithmetic challenge, is time consuming. We’ll be watching over the next days/weeks to see how people react to the overall implementation, the visualization of the map, the user interface within the challenges, the overall difficulty of the challenges, and the rewards.

So head on over to the site and give it a try. As always, comments, questions, and suggestions are welcome!

What 5th graders can teach you about design

Day one visiting the class of 5th graders that are participating in a Khan Academy pilot I’m looking around the classroom, and I see a student who’s exercise UI appears to be completely screwed up. The colors are all weird, and the background has turned black. My assumption: this is my fault.

Following standard operating procedure, I walk over and ask him how it happened. He looks at me like I’m a bit dim and says, “It’s easy. You just hit Control-Option-Apple + 8.” Boom! I’ve been in the classroom for 15 minutes, and a fifth grader teaches me something about my Mac that I didn’t know. This was a bit of a wake up call. There I was thinking of myself as an observer, but it’s clear that this is going to be a participatory learning experience. From that moment on, whatever remaining ego I might have had was shelved, and I was in full “I’m a student here” mode.

The kids proceeded to play, experiment, and learn using the interface, and most things went really well. Things got a little more complex as the students worked up to their grade level and needed to find help. We discovered a bunch of uncertainty about the right ways to get help. Some students couldn’t find videos; some didn’t want to watch a video because they weren’t sure if it would break their streak. A few students had mis-clicked the “Hint” button and lost their streaks. Not a disaster by any means, but a few fairly major issues that were impacting motivation and impeding good behaviors.

After leaving the classroom, Ben and I discussed a bunch of solutions and decided on a relatively small but obvious solution of reorganizing some of the UI to make students’ options when they got stuck a bit more obvious. Ben’s got some cool screenshots of the new exercise UI, so I won’t rehash that, but it was awesome to see some marked improvement when we went back in the next day. So awesome, that we are planning on spending as much time in the classroom as possible.

The whole experience got me thinking about just how harmful it can be, for both designers and users, when design is done in a vacuum.

What are we so afraid of?

XKCD:Airfoil

It’s funny, but nothing makes adults feel like frauds quite so much as a simple, well-articulated, challenging question to which we don’t know the answer, especially from a child. But what’s the big deal? The right response is so obviously right because it is choice that enriches everyone. I have this sneaking suspicion that the fear of being found to be wrong is what makes professionals of all kinds, but especially designers, afraid of interacting with and being scrutinized by users (customers/students).

In the case of the Khan Academy, there’s no room for that kind of thinking. Our success is not only dependent on our users actions, but also measured by their actual academic success. The abstract concept, user success as a measure of success of a design, is applicable to every piece of software out there, but revenue and profit tend to over shadow it. If dollars are increasing, most people are thinking, “Success.” We can’t hide behind that abstraction, and that means I can’t afford to be afraid of engaging students and teachers directly and frequently.

No one is completely immune to this fear, including me, but I believe the fear is driven by a misconception that the credentials that we build up during a career, expertise and credibility, are things with inherent value that you’ve earned (past tense). Expertise is really just the cumulative effect of all of the times you’ve been wrong but had the good sense to change your mind when a better answer was presented. Credibility is only valuable if it increases your chance of succeeding, because, speaking from experience, your credibility as a designer is likely worthless to your users.

Filters, like expertise, and tools, like credibility, should be valued in terms of what they enable you to do. They have no inherent value. The real value is created by the product of the design process, which means that optimizing for anything else is almost always going to be a waste.

It’s for the kids, so, handle it

The stigma surrounding “getting it wrong” is like a giant glue trap for progress. If I had to guess, I would say that, on average, we (designers) are wrong way more often than we’re right. I know it’s hard, because it’s hard for me too, but we all need to get over it.

Every time we fall head-over-heels in love with a particular implementation and avoid really engaging users to make it better, a kitten dies. More seriously, in my case, a student achieves less or a teacher becomes less effective. So the next time you set out to build something new, remember “the kids.” Be ruthless about removing as much ego from the design process as possible, because no one is enriched by defending a design that isn’t working. If your ivory tower design isn’t making them measurably more successful, embrace the fact that you’re wrong and work with your users to make it right.

This humble pie is delicious

From my first day working on Khan Academy stuff several months ago I’ve felt extraordinarily lucky to be able to work on a problem that I care deeply about. In September, I had a chance to speak with Sal and was humbled to think that I was playing even a small part in his vision for education. When I was offered the chance to work full time, I couldn’t stop telling people how lucky I was.

Well, I’ve only been at the Khan Academy offices for two days, and the pattern continues.

On Tuesday, Ann Doerr, one of our board members, stopped by to meet Ben and I. She brought us cookies, which, as anyone who knows me will tell you, is a surefire way to my heart. She was there because she wanted to make sure we had everything we needed for the new offices. She and Sal spent a few minutes discussing getting some cool white boards and painting the walls. Wednesday, she came back with a ruler and a contractor.

A few hours later, we went to meet with the team of folks at Google that is actively helping and supporting the Khan Academy, and in walks Peter Norvig (not to mention one of the original GMail engineers and a bunch of other super-star engineers and researchers). He tells us that he’s got a few concerns about the layout of the UI because he’d mis-clicked something when he was playing with it. After my brain stopped shouting, “OMG it’s that Peter Norvig,” it started shouting, “He’s using your stuff. Oh crap.” But there was no derision or condescension, just an honest desire to help us succeed.

So here are two people with the means to be spending their time doing essentially whatever they want, and they decided that it was worth some time to talk to us. And not just talk, but think critically about what we’re doing, make suggestions, and even jump in and help. Add that to all of the folks that are helping out with design, development, donations, and a few kind words now and then and one can’t help but feel simultaneously stunned and humbled. It’s enough to make any sane person wonder if the world’s gone completely crazy, but in a good way.

So on the off chance that this isn’t all some elaborate episode of Punk’d, I’ll offer a huge Thanks! to all of the volunteers, supporters, and benefactors of the Khan Academy. The challenge ahead of our small team is a big one, and I’m feeling incredibly grateful for all of the help and support.

From Ben's Blog: Khan Academy in the Classroom, Days 1 and 2

I’ve now spent two mornings watching 5th graders use Khan Academy in their classroom. The team has been splitting up to observe both 5th and 8th grade classes. It’s been thrilling to see and hear the reactions.

5th Grader: “I got 50,000 energy points last night!”

5th Grader: “I’m already on Adding and Subtracting Negative Numbers, where are you?”

Teacher: “Time’s up”
5th Graders: “Noooooo I want to finish my streak!”

Go check it out the whole post.

First iterations: the new Exercise Dashboard

Log on to check out the new Exercise Dashboard

We started with some pencil and paper sketches, set some design goals and created a concept, and this is what came out of the grape-press-o’-design.

I’m using the grape press metaphor for a reason. Designs are not finished when users get their hands on them; they are really just starting to mature. In other words, this design is grape juice, and by shipping it we’ve really started to add the various fermenting agents necessary to turn it into a fine wine. To be clear, we’ve been using the thing all week. We’ve solicited feedback from Sal and others throughout the process, but there is really no substitute for real-world use.

So here’s the short list of what we’ve tried to address with this iteration:

  • Make the knowledge map a primary means of navigating and understanding the exercises.
  • Refine/create a consistent visual language for a student’s progress relative to exercises across the list of exercises, individual exercises, and the knowledge map
  • Make the whole process a bit more fun/game-like

Here are things we cut from this iteration, but would like to include in the future:

  • Better reporting for teachers and students on their progress (especially over time)
  • Better/cleaner layout for the exercise content (actual questions and answers)
  • Better integration with the layout of the rest of the site (will be addressed by major site redesign effort, more on that soon).

To validate this design direction, or invalidate it for that matter, we are about to embark on an intensive pilot with a school system in the Bay Area. My plan is to document that process and learning here, in the open, so stay tuned. In the meantime, I welcome comments and constructive criticism, especially from frequent users. The most helpful comments are very specific (e.g. I was trying to this and it was hard because… or I really like the way this thing works because it made me feel…).

For the moment, it’s out there and aging on it’s own. Your feedback, and the feedback from the pilot, will help expedite that process.

Getting and using data

I generally avoid reading Internet comments, but Ben’s post about educators’ fear of technology made me look (Damn you, Ben!). Most of the comments on Sal’s original article can be summed up as “The current system works, you need to prove yours is better before we’ll listen to you.”

First I wanted to point out the “system” that all of these commenters are referring to only exists in the developed world. In the US, where we have plenty of well-trained educators, it’s easy to forget that those resources are extraordinarily hard to come by elsewhere. The ideas that are going to change and save the world from our current and future selves won’t necessarily come from the first world. Shipping thousands of tons of textbooks to remote parts of the world is not going to move information fast enough. The Internet is already revolutionizing communication and lending in these areas, why not education?

That point aside, “the system” doesn’t work the same for everyone. Lots of commenters focused on the uses of technology they’ve seen in classrooms. My mother happens to be an administrator, and former teacher, in the Yonkers Public School system. It is a very large, and relatively poor in terms of per-capita spending on students, school district. There have been plenty of initiatives that brought helpful technology into the classroom (like smart boards), but to this day teachers lack proper training and equipment to make effective use of that technology. Focusing on the existence of that technology in itself or on one teacher that has created an inverted classroom in one school in one subject is not proof that the old system is working.

So, let’s focus on proof. There have been plenty of studies and reports on the problems with the existing system, but I don’t care as much about those because they are only measuring outcomes. Optimizing for outcomes is what leads to policies like No Child Left Behind that, in practice, turn classrooms into yearlong test preparation courses.

We need to be optimizing process, and that’s what the kind of data that the Khan Academy is working towards gathering can do. Imagine real-time assessment data that tells a teacher or coach not only that a student is passing or excelling in a subject area but what specific concepts they’ve mastered and what concepts they’re struggling with. In this system, classroom time, a clearly valuable commodity, can be spent working on what students need the most help with instead of broadcasting a message that is, by definition, too broadly targeted to be helpful to every student in the room.

You want data/proof? Well, so do we. I just happen to think that we have a better chance of getting the data that really matters, and turning that into process innovation that really helps, than folks working within the textbook/lecture/classroom structure ever will.