Using rubrics to evaluate students and employees

This post is in response to a discussion last week at the Second Life Tech Soup Friday meeting.  The speaker, Gentle Heron, talked about employee performance reviews.  She gave tips about how to make them more pleasant; such as providing calendar dates, expectations ahead of time, and opening employee discussions.  I added a comment about how rubrics can help with these assessments…which lead to a suggestion that I do a presentation on rubrics this Friday!

A rubric is an assessment tool that helps score and outline performance expectations.

It got me thinking, ‘I bet most people haven’t been taught to use rubrics effectively!’  When people have an opportunity to rate themselves, and when they know that this rating counts, they take more responsibility to develop an awareness for what they do.  As supervisors and teachers, it is only fair to provide these expectations ahead of time. There is a very different feel when one is being judged, versus when one judges themselves.  When people are given the opportunity to critically self-assess, they become more aware of their job, and reflective and critical of their own progress.  A rubric can not only be an assessment too, but it can be an extremely effective teaching and training tool, too!

When I taught biotechnology to high school students, their grade was based on both my assessment, as well as their assessments.  We both filled out the employee rubric separately, then met together to discuss the scores.  Their final grade was an average between my assessment and theirs.

Here’s some tips:

• Start with a very clear rubric that outlines all the expectations for employee/student performance.  Sometimes, you may not know what all these expectations are, initially.  If you don’t, this is a GREAT opportunity to work with your employee or student to develop the rubric together.
• Give this rubric to people ahead of time.  Talk about what the assessment looks like.  For example, ask, “What does a score of 5 look like in terms of attendance?  What about a score of 3?  …and 0?”  Begin these conversations now, so that there are no surprises.  That way, the rubric categories set a standard, and will not be taken as a personal affront.
• Evaluate mostly formatively and occasionally summatively.  
  • Formative assessments mean that you and the employee/student look at performance periodically, and reflect/revise as you go along.  Think of a chef when they’re cooking an elaborate stew.  When they are constantly tasting the soup, adding spices here and there, adjusting things during the process, they are doing formative assessments.
  • Summative assessments are at the end.  Personally, I do not think that these are as useful, but yet the world (starting with education) has somehow used them as standard.  Summative assessment is like the final taste test for the food contest.  The food is all cooked, finished, and there’s no going back.  This is really tough on an employee/student – and if you think about it, where is the opportunity to learn?
• Provide spaces between evaluations to reflect, revise, and adjust not only employee/student performance, but the rubric, itself.
  • As a supervisor or teacher, be a “guide on the side,” and mentor – encourage, ask questions, and push, but don’t dictate.  From my experience, people “own” their work and their self assessments when they not only understand what is expected of them, but also that their voices, their input counts.
    • Start by asking the employee/student to justify their scores.  This will give you a good idea of whether their interpretations of the rubrics match your expectations.
  • Assessment and evaluation should be a conversation with clear expectations and understanding.

Here’s a list of places to go for rubric designs and example templates.

iRubric – This site provides starter templates for you to design employee rubrics.  I would start with a generic template, then pull out the job description (or your learning goals), and then customize from there.

Rubistar – This is a great rubric website for teachers.  Similar to iRubric, you can take a pre-made template and customize it to your needs.

Cooper’s Rubric Presentation – Talks more in detail about rubrics, and includes several different types of employee performance rubrics, as well as outlines the steps on how to develop one.

You can also Google “Employee performance rubrics,” “Student rubrics,” “Music rubrics,” etc.  to get a base template to begin.  Then, customize it to your needs.

Do you have questions about rubrics?  Comments?  Please ask away and I will try my best to answer!

Next up on rubrics:  Creating them.

What’s a Program Evaluator?

In addition to curriculum design and teacher mentoring, I am also a program evaluator.  It did not occur to me that most people don’t know what program evaluators do, until my mother asked me about it.  I floundered over my explanation at the time, since I was caught off guard.  However, after pondering for a bit, here’s a more elegant (and detailed) answer:

A program evaluator analyzes data collected from a program to see whether or not the program is effective in doing what it is supposed to do.  

The evaluator’s data collection and analysis is in the service of answering one very big overarching question:

We then use more specific, detailed questions to outline the actual evaluation itself. These questions will depend on what the program stakeholders want:

• What are the components of the program (activities, processes, people), and how do they work together?
• What components are working well?  What components are not working well?  Why?
• What is the impact that the program is having on the stakeholders that are involved?
• Is the program fulfilling its mission statement?  Why, or why not?
  • What evidence can we collect to say that a program is or is not working?
• How can the program be improved?

Of course, it can be more complicated than this, and there are many different program evaluators that are out there; each with their own different styles.

I, personally, take on the role of a critical friend, rather than judge and jury – so in most of the programs that I have evaluated, my analysis and reports are about helping people improve their programs so that they can better serve everyone involved.

It’s a rather gratifying experience, since my job has allowed me to work very closely with people as we plan out the type of data I will collect, what information I can provide to the program in terms of understanding how their programs work, suggestions for how they can best accomplish their goals, and how they can improve.  My data analysis has also been used in publications!

Although program evaluation can be an extremely rewarding experience, you do need quite a bit of training.  Evaluators should have working background knowledge of the programs that they evaluate (for example, I have an emphasis on STEM education), and they also have special training in evaluation, social research and data analysis.  I would also add that they should be a people person – interviewing skills are important for the job!

For more info about program evaluation, start here.

Do you have questions about program evaluation?  Similar to education, I can talk on and on about it!  What else would you like to know, hmmm?

Online Education: Beyond the technology – Part 1*

And now a post that is very near and dear to my heart – online education!  I realize that Virtuosity touches upon a variety of topics, but at the core, my passion is teaching teaching.  Why?  Because if you can teach people how to teach, you’ve just empowered the world.

Anyways, when educators discover that I design and consult about online curricula, they start to ask me a lot of questions, or they tell me about their experiences in online education.  One of the most common situations that I’ve run into (especially with virtual educators) is: I have my students taking my online course, but they’re not (self-directed, motivated, performing, getting higher grades, participating, interacting etc…) as much as I thought they would in this environment.  People will also come to me and say, “Oh, I tried teaching online, and it didn’t work.”

Okay, let’s tackle the big fallacy underlying these two very common situations:

There is an assumption that technology or online (anything) is the silver bullet to teaching.

With the introduction of the Internet, along with lowered costs for personal computers in the 1990’s (Reiser, 2001; Harasim, 2000; Kapp & O’Driscoll, 2010), educators began jumping on the bandwagon – touting that the Internet’s capability to connect people to a world of resources would revolutionize education (Reiser, 2001).  Even now, we can still easily find articles on game theory, gamification, MOOCs and other emergent technologies that claim that a new educational paradigm is upon us – a transformative force that will move our society into a knowledge-based, information age.

This is all hype!  The Gartner Hype Cycle is wonderful model to explain the typical trends that emergent technologies make over time in our society.

Taken from: http://www.ask-force.org/web/Discourse/Linden-HypeCycle-2003.pdf

As you can see from the figure, when technologies enter the scene, they do it because of “inflated expectations” (Gartner, 2003, p. 5) of what a technology may do (whether it is the promise of enlightenment, educating a world of people, or the instantaneous motivation of a classroom full of freshmen) – everyone jumps in, thinking that this will save the world.  Afterward, when all the pixels have settled, and we compare digital education to face-to-face traditional methods, we find that on average, most studies show no difference in student performance (Bernard, Abrami, Lou, Borokhovski, Wade, Wozney, Wallet, Fiset, & Huang, 2004**).  In fact, once everyone has jumped onto the bandwagon, we begin to see a windfall of literature that talks about how the technology sadly fell short of expectation.  Educators end up in the ‘trough of disillusionment,” where not only is technology is not what it was cracked up to be, but also, may be even worse than what we had before!  Unfortunately, there have been many occasions where educators and students stop here – embittered that technology did not fulfill its (false) promises to reform.  We have seen this happen several times with online education, and can follow this curve in the literature.  I’ve read about the hype cycle regarding Second Life, and now on MOOCs, as well (2012 was supposed to be the “Year of the MOOC“).  I would argue that this is actually the reason why many reform measures fail to work – we don’t hold out for the “slope of enlightenment” phase, where we actually learn when and how to use technology appropriately!

Just how do we arrive at the slope of enlightenment, you might ask?  (Well, maybe you didn’t ask, but I shall ask it for you).  First, we must reevaluate our assumptions about technology in education, and reconsider:

1. You can lead a student to the Internet, but they still may not learn.
   • We must debunk our assumption of the “Net Generation” – that young people (I cringe at this term) are simply born with a keyboard in their hands and that they will not learn any other way.
   • The other assumption to this, is that the Internet is self-teaching.  This is largely due to Sugata Mitra’s infamous “hole in the wall” study*** – a study that has been critically examined, and unfortunately, myth busted.
2. “Insanity is doing the same thing over and over again, and expecting different results.” ~Albert Einstein
   • To clarify what I mean by this – I refer to several times when I have been invited to classes in Second Life, only to sit in a virtual chair, with virtual classmates, looking at a virtual board – with Power Point slides.  To use technology to do the same things we’ve always done sort of defeats the purpose.  Studies have shown that when instructors have adopted technologies in their classrooms in this way – to use technological superficially as a “new way” to do “old things,” it doesn’t stick or become incorporated effectively within their programs (Lankshear & Bigum, 1999).
3. Online teaching requires online pedagogies that are very different from face-to-face teaching.
   • Teaching online using exactly the same pedagogies as face-to-face teaching is like using Power Point to teach a yoga class.  It doesn’t work.  Instead, we must develop, understand and use appropriate methodologies that are aligned with our teaching philosophy and learning goals.
   • Good teaching is good teaching, regardless of where or how you do it.  Good instructors adjust the “how” part, depending on “where” they are and what they are teaching!

So, I mentioned three assumptions.  Although they may seem small, each one is like opening a box of Sees Candies chocolates – full of wonderful surprises.  In future posts, I intend to unpack some of these assumptions for you!  In particular, I will address point #3, because this last one is, what I would claim, the Achilles Heel of the online teaching world.  When online teaching is done incorrectly, the results can turn both teachers and students away from it altogether.  One of my hopes as a curriculum consultant is that I can prevent that from happening!

Stay tuned fellow educators, grad students and intellectuals – I have a lot more to share on this subject.  However, if you have burning question for me about it, or a specific topic you’d like me to blog about, please leave it in comments, or contact me.  I’d be more than happy to help you!

 

Notes:

* I am a huge believer in accessibility to all audiences.  Therefore, I have tried my best to link you to credible open sourced documents.  This has resorted in a simplification to my references – open sourced links will not be listed in the reference section, since you can click on the embedded links in the blog.  However, if you are eager to learn more, or would like further citations, please let me know.  One advantage to being in the middle of my comprehensive exams is that my brain has been thoroughly marinating in this topic for several months!

**There are many other studies I can cite that show this.  However, Bernard et al.’s meta-analysis is one of the most comprehensive reviews on the literature regarding online  versus face-to-face instruction.  Let me know in comments, however, if there’s others that you’d like to see!

***Check out Mitra’s talk, here: 

References:

Bernard, R.M., Abrami, P.C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., Wallet, P.A., Fiset, M., & Huang, B. (2004). How does distance education compare to classroom instruction? Review of Educational Research.

Harasim, L. (2000). Shift happens: Online education as a new paradigm in learning. The Internet and Higher Education, 3(1), 41-61.

Kapp, K.M., & O’Driscoll, T. (2010). Learning in 3D. San Francisco, CA: John Wiley and Sons.

Lankshear, C., & Bigum, C. (1999). Literacies and new technologies in school settings. Pedagogy, Culture & Society, 7(3), 445-465.

Reiser, R. A. (2001). A history of instructional design and technology: Part I: A history of instructional media. Educational Technology Research and Development, 49(1), 53-64.

Catching Butterflies

She closed her eyes

and opened her mind;

released her thoughts

and let them find

the fire in the butterflies.

Its seems at the start of every large writing project, I find myself at the edge of a cliff with a large net.  The sky is full of colorful butterflies – their wings dazzle and catch in the light while they fly erratically – teasing, nearing, then flitting unpredictably away, while luring me from my safe space, daring me to step off that safe cliff rock to fall into the dark abyss.  I am mesmerized by their brilliance, as wings collide and dance in front of me.  I only need to catch a few – but they must be specific kinds!  So I watch, and try to shake myself out of that overwhelming stupor – the confusion of having so many ideas and thoughts spiral around me.  I spend days there, focusing, concentrating on that single butterfly that I must catch.  Sometimes, I find them in my net, and am fooled – a stray petal or leaf, but not the butterfly I want.  At other times, I catch so many, and I must only pick a few.  Their brilliant colors confuse me, and before I can get out my collecting jar, they have all flown away.

Find me today, on the edge of the cliff.  I’m catching butterflies, again.  I hold my net and my jar, eyes locked onto the swirls of color… I just need a few.  Only a few.  Just a few to light my way.

Then, I will be ready to jump!

 

The Many Faces of Battle

One of the philosophies behind poststructuralism is that individuals are made of many different individuals – we choose to be different people, depending on our environment and who we are with.

For example, when we take on our virtual masks, we may present ourselves, or aspects of ourselves in very different ways than we would represent ourselves physically.  Even physically, we are different people when we are say, at a board meeting versus in a family reunion!

As of late, I have kept much of my private battle with academia behind the scenes, but it has been a struggle.  This face has been one that I have kept hidden until now.  As a graduate student, I have been working on my comprehensive exams – a rigorous evaluation period where students are asked to write very deeply and knowledgeably about specific research topics.  My topic just happens to be online learning and education!

Since April, I have been actively working on my comprehensives, or fondly known as “comps.”  The process typically lasts from six months to a year for graduate students.  During this time, I’ve learned so many things – not only about my subject matter, but about myself.  As I continue on my journey through comps, I wanted to take this moment to pause, and reflect on some of the things I’ve learned – they are things they never tell you about graduate school, but I think that a lot of PhD students struggle with:

1. You don’t know enough, and in everyone’s eyes, you are a rookie.  It doesn’t matter how many years you’ve spent in your field, or what you’ve done – when you begin your journey in graduate school, you begin again.
2. Get used to feeling stupid.  Everything you believed or knew when you started will never be or look the same – especially when you read, discuss and write about other peoples’ research, theories or ideas.  They will always change your own – making them more three dimensional, and oftentimes, far more complex than you thought!
3. You are always unsure, because you are learning about what you don’t know, and what you can do about it.  It’s a very humbling experience to realize, almost every day, that you don’t know most things.  …and just when you think you know something, along comes someone else who knows even more, and who will challenge you to stretch your mind, and your thinking.  Grad school isn’t a place for close-minded people, or for people who think they know everything and are personally insulted when they are told they do not!
4. You spend a lot of time inside of your head – it’s like you’re unraveling tangles upon tangles, and there’s some days when you don’t know where one tangle began, and the other one ended.
5. It can make you irritable, and insecure.  Oftentimes, I’m left to wonder whether I’m competent to do anything at all!  But yet, there’s something about that insatiable thirst to understand, and that drive to do better.  I want to change the world to make it better – that passion keeps me going, even though there are times when I throw pencils and want to quit.  There’s no quitting.
6. You must look in the dark and face your fears – laziness, hesitation, procrastination, ignorance… it’s not fun to constantly feel stupid, and so, there’s times when we must resist the temptation to do something…anything… to feel like we know something again!

I have friends who do not understand why I hesitate – especially because they know that I tend to be a very blunt, forward-thinking person.  Yet, when it comes to comps, there are times when I have felt scared, because there is a very good chance I could fail.  I tell them that comps are like going into a pit of snakes every day.  You dread it.  It’s not easy, it’s slow, and it’s not like anything you’ve ever done before – facing your stupidity at every sentence, and they expect you to be a snake charmer in the end.

Yet, after this third go – this next time around… the snakes… they don’t seem so bad, anymore.  In fact, some of them are some of the most beautiful creatures I have ever seen.

 

 

One Billion Rising in Second Life

Today, I got to unveil my first 3D art project, “Transgressing Through the Looking Glass” at One Billion Rising in Second Life!  What an amazing journey it has been personally for me, as I wrestled with difficult concepts, figuring out ways to not only express myself, but also…how to reach out to others, too.  My display is in OBR Drum Region.  You can take the landmark to get to OBR, and walk about.  There’s lots and lots to see!

http://maps.secondlife.com/secondlife/OBR%20Rise/45/28/25

The art is only up until tomorrow, then it all comes down.

My flickr feed, just to give you a taste:  https://www.flickr.com/photos/128448694@N08/

Transience….

The sign before walking up the path to the exhibit.

Tuesday’s Talk on Pathogens

This Tuesday, November 18th at 6:30 pm SLT, I will host a talk at Meeting of the Minds in Second Life, where I will explain the different kinds of pathogens that have plagued (haha!) human kind over the ages.

Everyone is invited, although you will need to access the group (you will get an invite when you teleport to the sim).  Robust discussions will follow!

Location: http://maps.secondlife.com/secondlife/Dreamworld%20Sea/11/62/25

 

 

Getting on my soapbox…

Doctors Without Borders is one of the very few organizations in West Africa trying to fight and contain the Ebola outbreak. As of October 29th, there have been 13,567 identified cases, and 4951 deaths. As of October, there are only 15 operational treatment centers (1,047 beds) in West Africa (only 2 are in Guinea!!!!) for THREE countries. (http://www.nytimes.com/interactive/2014/07/31/world/africa/ebola-virus-outbreak-qa.html)

Although the World Health Organization plans for 41 centers, poor infrastructure and lack of medical staffing are HUGE issues (think of the cost for equipment, supplies, etc). If this outbreak goes on with out interventions, an estimated 1.2 million people in West Africa will be infected by January 20th.
Granted, this is NOT the alarmist pandemic that the media would like us to believe. It really isn’t going to be a big issue here in the U.S. (reading fact sheets from WHO, CDC, and following Eboladeeply.org puts things in perspective). BUT, if the outbreak isn’t contained in West Africa, it will continue to persist and be a threat to the world (although this isn’t the focus of my argument).
On the humanist front, WHY ISN’T THERE MORE BEING DONE?! 9 billion dollars was pledged to Haiti on a global level to help the country rebuild after the Haiti earthquake.
Yet, it took me a bit of digging to even FIND an organization involved with the Ebola outbreak (http://www.doctorswithoutborders.org/).
Why?
Let’s stop freaking out, panicking and shutting our doors in this “it’s over there, they can die, but WE MUST SAVE OURSELVES” mentality. How about we face the issue: People are dying from a horrible disease, and in West Africa, they need help and aid to contain it.
Let’s not make this an issue of shutting our borders and letting people die. Instead, let’s make it an issue of helping out and making sure this disease doesn’t come back!

Finding the ovaries…

I don’t need balls.  People expect balls.  They react to balls like a scripted sex scene out of a badly directed porn series – thoughtless, automatic, and it never goes beneath the surface to what counts.

I need ovaries.  I need the ovaries to continue my search for language and expression that creates; for meaning that challenges and provokes thought; for words that empower.  I crave for the words that tear past the ripples of normalacy, burrowing deep – drilling past bone to lodge into the soul where it lives, stirring, fluttering, never allowing its human to ever rest again.  Words that drive the sleeper to action – to stand for what one is fighting for, to question and investigate.

They are the words to the other self, created by a language that is created when they are spoken.  I need the ovaries to stand up and find this language, in spite of those blinded by the other truths, who will never understand their meaning.

Come for my Tuesday Talk!

On Tuesday, October 21, 2014, in Second Life, I will be presenting: Using the F word to Confront Gender Inequity: A Historical Evolution of the Word “Feminist.”

This talk discusses some of the challenges humans face with gender inequity.  In order to start productive conversations and plans of action to confront social injustices, one needs to break past stereotypes and look deeply and differently into the normalizing patterns in society that oppress.  This talk is not about man-hating, Feminazis, or bra burning (that was so yesterday).  Instead, it is a brief history on the word “feminist” – how it evolved, why it started, who started it, and what has happened because of it.  Also… a question of… where do we go next?

Only through deconstructions that looks critically at decentering and difference can we start to position ourselves in ways that are not only empowering, but also moves us towards creating the changes we want to see in an equitable world.  This talk will be the first in a series of lectures on the topic of empowerment and  voice.  I hope you can join us!

The audience is invited for discussions afterwards.

This talk will start at 7:00 pm SLT (10:00 EST) here:  http://maps.secondlife.com/secondlife/Dreamworld%20Sea/47/66/25

Hope to see you there!