Wednesday, August 28, 2013

Sebastian Thrun: Update on our SJSU Plus Summer Pilot

We have some exciting news to share on our SJSU Plus Pilot. To remind everyone, with the encouragement of California Governor Brown, Udacity teamed up with SJSU to deliver for-credit college classes online. A key objective for this partnership has been to bring college education to students who are presently under-served and left out of higher education. 

This summer, we ran the second instance of our pilot. While in the Spring, we actively sought out underserved high schools from low-income areas in California, this time we simply opened up enrollment to anyone. As predicted, with 2,091 students who enrolled, we mainly reached students who would not ordinarily attend college. Only 11% of the summer students who took the for-credit courses from SJSU were matriculated students in one of the California State Universities. 71% of our students came from out of state or foreign countries. And while the total number of high school students went up, their proportion in the total student body went down.

Hot off the press, here are the latest results for the summer pilot. For comparison, we also provide rates of students who received a C or better for the Spring and Summer pilots, and the comparable rates for on-campus classes at SJSU averaged over the past six semesters.

In short, pass rates are up and match more closely those in SJSU on-campus classes.

We did change a number of things. A good fraction of the content was re-done between spring and summer. We used data from the spring to understand what parts of our content worked, and what needed to be improved. We added hints for challenging exercises, and we added more course support staff to assist with online discussions and communications. We also changed the pacing methodology, informing students earlier and as part of their course experience when they were falling behind. And reaping the benefits of all the hard work in the spring when they were busy preparing all the content, SJSU’s professors spent much more time over the summer interacting with students. Undoubtedly, these changes had an impact on the student experience and success rate.

Another way to achieve high pass rates is to be highly selective in the student admissions process. Elite private institutions are masters at picking the very best students, and consequently their graduation rates are amazing. We wanted this program to be the opposite.

One key difference between Spring and Summer was that we opened our Summer session to everyone. This led to a substantial difference in student body. Among the student body, 53% reported that they already hold a post-secondary degree (5% Associate, 28% Bachelor’s, 16% Master’s, and 4% Doctorate). Only 12% of the students had a high school graduate diploma or equivalent, and 15% were active high school students. This is very different from the Spring Pilot, in which approximately 50% of the student body were active high school students, and the other 50% were matriculated SJSU students.

And interestingly, college credit was not the leading motivation for students to take these courses for credit. With love of learning, career advancement, and lack of options all part of the equation, we found we were serving a broader need:

Yet with these new results, we still aren’t there. There remains so much more that needs to be improved. The summer pilot was the second iteration of a new approach. To all those people who declared our experiment a failure, you have to understand how innovation works. Few ideas work on the first try. Iteration is key to innovation. We are seeing significant improvement in learning outcomes and student engagement. And we know from our data that there is much more to be done.

We know that students learn at different speeds. This is particularly the case in the mathematical sciences, where it just takes a while to really understand certain concepts. Rushing students through a timed curriculum with a pre-defined pace cannot be the best way to achieve lasting success. In our remedial math class, we only gave students a single chance to pass various exams. If they even failed the first midterm, they failed the class. On campus, multiple chances are offered. There are clear opportunities to rethink assessment as a whole, especially as we open up new pacing options.

It is important to acknowledge that we are entering uncharted territory. I have been fortunate to work with some of the very best innovators in the world. I have learned that innovation requires a clear vision, many iterations, and a willingness to learn and improve. I believe we are on a path where we can break down many of the barriers to access in higher education. But this is just the beginning of this path. I am excited to see where the next round will take us.

UPDATE November 15, 2013:

About our experimental methodology, specifically with regards to our Spring Pilot -- To clarify, student participation was voluntary, and the experiment was performed with IRB approval and in accordance with IRB guidelines. Thanks to a generous gift from the Bill and Melinda Gates Foundation, participation was free of charge for all students. We involved the National Science Foundation, WASC, UC Berkeley, and SJSU’s own IRB in setting up and evaluating our experiments, and we had faculty from Stanford’s School of Education on our team to guide us with pedagogical choices. Udacity did not receive any public money for this work.