How to measure how well the MBTA (or your instruction) is working

tmap2-croppedAccording to Walk Score, Boston ranks #3 on their list of transit-friendly cities (see the rankings), with its Transit Score falling in the “Excellent Transit” category. If, like me, you rely on the MBTA and routinely watch multiple over-crowded trains go by before managing to squeeze onto one or spend over an hour on a five-mile commute, you might be scratching your head over how Boston could achieve such a high ranking. How exactly did they measure the quality of public transportation? 1 Have the folks at Walk Score ever ridden the T?

The broader question raised — one that an assessment of public transit shares with an assessment of your own instruction — is: How do you measure how well something is working? Whether you are conducting a formal assessment or informally trying to gauge how your instruction is going, there are a few things to keep in mind starting, perhaps most importantly, by defining what success looks like.

Defining Success:  Is an “excellent” transit system one that has bus stops every 50 yards? If so, then Boston certainly deserves its high ranking. But some of us might define success by how efficiently we can get to work and back, which stopping every 50 yards tends to hinder. Success in teaching is often defined by student mastery of learning outcomes. These can include the skills and knowledge often measured by problem sets and exams, but also attitudes about the course material which require a different approach to measurement. For some classes or programs it can be helpful to invite other stakeholders, like potential employers, to help develop your definition of success. Success can also go beyond student outcomes and may include outcomes for you or for your TAs as well.

Measuring Success:  Once you have your definition of success, it is time to devise a way to measure it. In devising your measure, remember to consider multiple sources of evidence. The transit scores that led to Boston’s high ranking do take multiple criteria into consideration, like distance to stops and frequency of scheduled service. However, they do not consider other criteria like adherence to the schedule, overcrowding, whether people actually make it to their destination, etc. This would be similar to assessing your course based only on the syllabus. You may have a great plan for your class, but you also want to look at how the implementation goes and what the outcomes are. When considering what evidence to use in assessing your instruction/class, it is worthwhile to examine the plan, the implementation process, and the outcomes.

Also, remember that the most useful “measure” may not always be a number. Going back to the MBTA situation: although a ranking of three or a Transit Score of 74 might make the data appear nice and simple, it doesn’t tell the whole story or help guide improvements. Observing the overcrowded trains or talking to people who rely on the MBTA for transportation would provide more useful data. Similarly, examining student work or talking to students about their experience may be just as important as looking at final exam scores. Mud cards are a tool that can help you assess a class session or activity to guide your instruction in the future.

Refining: There is no perfect measure or perfect instruction. Continue to refine your instruction and your measures of success. The folks at Walk Score do recognize there are limitations to their measure. They began by focusing on the theoretical availability of transportation. As they refine in the future, they may focus more on measures of travel time. If you begin to try new instructional strategies, the focus of your assessment may change. It may be helpful to target certain strategies or learning outcomes in your assessment rather than trying to measure everything at once. For example, you may want to focus on student engagement in one semester, using observations of student participation, completion of assignments, and feedback on student surveys. Then, as you tackle some areas, you may want to refine the measures or focus on measuring other aspects of the course.

Deciding how to measure your success in teaching can seem like a daunting task. The Teaching and Learning Lab offers workshops as well as individual consultations for the MIT community to assist with this process as well as determining what to do with the results. Feel free to contact us as you consider your needs for course assessment!

1 Boston’s rank is based on a computed “Transit Score” of 74/100. This score is derived primarily from the average distance to a transit stop, the type of transportation (train, trolley, or bus), and the number of scheduled runs. (Details on Transit Scores)

(“Excellent transit?” from Anne Marshall / cc by-nc-sa)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s