Virtuosity 11.11

Where words become worlds…

Archive for the category “Education and Program Evaluation Topics”

Moving Forward

I took a long break from blogging and my online community to concentrate on the final steps of dissertation defense. This weekend, I graduated with my Ph.D. I could not have accomplished this without all the help from the Builder’s Brewery, my advisors, my family, and my friends. People have asked me about what I will do next! There really is so much to do, and now, I’m in a very good position to do it all. I’ve listed bullet points here of my five-year plan, but this is in no particular order…

  • Present my work to the Builder’s Brewery community – once my thesis is released publically, I will present a lecture to the communities in Second Life to share the foundation for my learning theory, which I call Interactive Spatial Learning, or ISL.
  • Attend the Connected Learning conference in Boston in August, and the Association of Internet Researcher’s conference in October. I present parts of my thesis to these conferences.
  • Publish ISL in at least two journals.
  • Begin my work at Paragon Learning Research Group as both the CEO and the STEM Education Director. This work is a culmination of all that I’ve learned from the Builders Brewery in combination to attending to the needs of teachers here in my local district. Thanks to No Child Left Behind, many students no longer have solid foundation in science, technology, engineering, and mathematics (STEM) because the focus was on literacy and mathematics. Unfortunately, STEM cannot be taught without context and “doing” of things. My goal is to create a digital support system to help with this work, and to help spread STEM education everywhere and to everyone.
  • Continue the direction of Virtuosity, but perhaps thinking of ways where I can make this blog useful. I’ve learned so much from listening to peoples’ stories. My goal now is to continue sharing those stories, with the hope that they are useful.
  • Continue my mission to change the world in positive ways by pushing the different ways we think about learning and education.

Thank you for those of you who have supported me. I would not be where I am, accomplishing this very large life goal, without your help. This was one of many large mountains that I need to cross. Hopefully, I hope you will join me on this journey!

Also, if I can be of any help to you, please contact me. I’ve learned by helping others, we also help ourselves, and the world shines just a little brighter through these partnerships.

Advertisements

Dissertation updates and other things

I’m sure there are people who have been following my work are wondering where I’m at now, with my dissertation writing. These past few weeks have been extremely busy for me as I focus on making the deadlines for graduation in May.

As of today, I’ve written Chapters 1-6 of the 7 chapters of my dissertation. I need to wait until I’ve completed the formation of my theory, which makes up Ch. 5-6 before I can finally write about Ch. 7. Every chapter undergoes several revisions before they are good enough to submit to my committee. Right now, I’ve finished the first draft versions of Ch. 1-3, 5, and 6. I’ve completed the second draft revision of Ch. 4.

So, what’s next on my docket right now is working on the second draft revisions for Ch.s 5-6. Both of these chapters summarize my findings and learning theory, and there are a lot of holes I have to account for in them. My theory constructs are still very tentative, so now, I must work on making them more concrete and grounded into the data. This week, I dive back into the data, with a focus on:

  • Developing themes from my interviews
  • Member-checking with members of the BB community by making sure that my observations and interpretations are accurate with my key informants
  • Polishing up my code book
  • Developing definitive constructs for my theory
  • Summarizing the findings from my survey

Right now, it feels like I’m on a treadmill, and I just have to keep running to make things in time. Apologies if I haven’t surfaced in a while, but rest assured, that I’ll be sharing bits of my data in the next few months!

Wishing everyone happy holiday productivity!

 

Week 4 – Presentations, Proposals, and Conferences, Oh MY!

My apologies for not being able to keep up with my blog posts! With November 30th looming closer on the horizon, I feel like my work is funneling down, and I’ve had to quicken my pace just to keep up!

Last week was the first of many presentations that I am doing on the learning model derived from my dissertation study. As a hybrid between the folks at the University of Rochester’s Learning in the Digital Age (LiDA) community and a Zoom session with over 25 people from the Builder’s Brewery community in Second Life, it was an amazing collaborative experience! I walked away with many ideas as I continue to clarify my language, strengthen my arguments with evidence, and make modifications to my model. The exciting thing is that so far, it seems that I’m pretty much on target with what people are saying, and that my interpretations are in agreement with what people perceive. This is imperative for the qualitative work that I am doing!!!

This week, I’m completing the final edits to a manuscript (accepted for publication!) from my previous work with the Rochester Institute of Technology’s COMETs program. I’m also submitting two (of four intended) conference proposals about the findings from my dissertation. Additionally, I have one more full chapter to work on (yikes, it’s the big finale chapter!) before going back to do total revisions. So far, things look good, but it’s always going to be a scramble to get everything written in time!

Hopefully, I’ll be at a point where these updates come more often.

Hope you are having a wonderful week!

LiDA talk

PowerPoint Week

The infamous “Wall of Text” power point slides are over. This week, I’ll be talking a bit about how to make a powerpoint presentation, and offer some advice on how to get your story and your slides up to speed.

In the meantime, I start this topic off with some expert advice from Dr. James Hayton’s PhD Advice Website. He provides some very short and sweet pointers to keep in mind when making your slides in How to design outstanding power point slides.

2000px-microsoft_powerpoint_2013_logo-svg

Transcription Tips Tutorial

Transcriptions are an integral part of the research because they provide a written record of the audio from your interviews and focus groups.  However, without the right tools, they can be VERY time-consuming.  The two biggest tips I can give you are: make the cleanest recording you can and use a transcription program to help you. Here’s how:

Find a quiet space, and have multiple recorders going.  Don’t record in a coffee shop or a place that echoes.  Any place with a lot of background noise will give you the transcription from Hell because it’ll be hard to hear what your interviewee is saying.  For a typical interview or focus group session, I usually have at least two recorders going at the same time.  I’ll use my iPhone, my computer, a recorder, and when possible, a microphone.  For focus groups, I’ll put the recorders in different parts of the room.  I prefer to video record when possible as well so that I can see facial expression and body language.  However, to be able to visually record, you need to check with your interviewees and your RSRB to make sure you have permission.  Even with audio, please ask your interviewee before you record.

CLAP before you record your metadata.  If you are recording video, do this in front of the camera.  This will cause a spike in your audio files, and it will make syncing all your files together MUCH easier.

ALWAYS record metadata.  You can start with something like this:
Today is (date), we are doing a focus group interview at (location), it is (time), and with me are: (ask each person to say their name clearly, and give a brief intro that will help you identify their voice and name on the recorder)

Take fieldnotes when you can.  Although this will depend on the nature of your interview.  If I’m doing focus groups, I will have my computer up, typing notes as people respond to interview questions.  This is because when I type my notes, people actually pay less attention to me, and more attention to the others in the room – which is what I want.

However, if I’m interviewing one-on-one, it will depend on who I’m interviewing.  Sometimes, having a computer or notebook up may make the interviewee uncomfortable, and you won’t get spontaneous responses.  It will really depend on the situation.  If you’re in a situation where you can’t take notes during the interview, then make sure you jot things down as soon as you can – so that your memories are fresh.

When the recordings are finished, sync all the files using an audio editor program, such as Garageband or Camtasia.  Remember to line up your “clap spikes,” so that all your audio is synced.  These editor programs are REALLY useful for taking out background noise, too!

Import your edited file into a transcription program.  I swear by Inqscribe, which I love because everything is in one program, you can speed or slow the recording, control the start and stop with the tab button (instead of a foot pedal), and you can tag your file with timestamps (see below).  ALWAYS tag your file with timestamps.

Make time: It will take about an hour to transcribe 15 minutes of audio (from a clean recording).  I transcribe in blocks of time – because you will burn out after a few hours!

Tag your file with timestamps.
Save time on the first pass through: Depending on the purpose of your transcription, sometimes you can just paraphrase and timestamp, while relying on field notes.  Timestamps will allow you to go back into your file and quickly get to where you need.  I timestamp periodicially – especially before important things have been said.  Also, if something is inaudible, just type “inaudible” in your transcription to save time and move on.

Later, if you are doing discourse analysis, you can go back slowly over everything to include the transcription notation (transcriptions may take several passes – depending on how you will analyze these data).

Add dates to your file names
Label your files with the interview date (ie. 19Sep16 – Gidget Interview).  Also, if you can (some places allow for this), a description of the interview – ie. who was interviewed, where, and what it was about.  Keep these data files in a place that is secure.  Personally, I do not use Google for confidential data.  Instead, I use a Box account through my university which insures privacy and security.

Happy Transcribing!

cbs-listening-post-transcription-1941

 

DIY Whiteboard Tutorial

I learned this little trick from my son’s 3rd grade teacher.  With just a pack of sheet protectors and paper, you can make an entire class set of “whiteboards” to use in the classroom or for your own personal use when studying (I’ll talk about that part next week).

You only need three things: paper, sheet protector and whiteboard markers.

IMG_2542 (1)

Really.  That’s it.  Put the paper (and this can be blank paper, lined, graph, etc) into the sheet protector.  Write on top of it with a whiteboard marker, and ta daaa!  Done.

  • To make the sheet protector last longer, I’d suggest using an old sock or rag to wipe as opposed to a piece of tissue.
  • Put a piece of hard cardboard in the back, and you can write on the surface anywhere.
  • Get a metal clipboard, clip the page protector onto it and you have a portable magnet board, too!

Do you have other variations?  Thoughts?  Ideas?  I’d love to hear from you!

Next Tuesday, I’ll show you how to use flash cards in conjunction with your white board to help you actively study.

 

 

Tuesday Tutorial – Evernote for Lectures, Snapshots, and Scans

Hi Everyone!

I have been doing small talks at the Non-Profit Commons in Second Life, where I have had the pleasure of connecting with a lot of passionate people.  This Friday, I’ll be interviewed by blog radio host, Marie, on her Talk! with Marie show about my work.  In the spirit of this upcoming show, I wanted to highlight an excellent suggestion that she gave me – that people would be interested in tutorials that show what I use in my grad school work!

Evernote is one of my first tools, because it is not only an electronic journal that keeps everything for me, but the paid subscription means that it makes everything SEARCHABLE!  Even my hand-written notes!

Below, I present a quick tutorial on how to get those  hand-written notes, and the notes instructors put on the board, into your Evernote in a fast and easy way.  Look on, and prepare to be amazed!  (Well, at least I was, when I first tried this trick!)

Using my nifty Camtasia program, if you view this on Youtube, you can actually click to the marked chapters – which might make viewing more pleasurable to you, if you have a short amount of time.

Enjoy, and let me know if there are other sorts of tutorials that you might be interested in seeing!

Writing Rubrics

As I mentioned in last week’s post on rubrics, a rubric is an assessment tool that helps score and outline performance expectations.  Using a rubric helps by:

  • verbalizing expectations for performance
  • standardizing these expectations
  • providing benchmarks for assessment
  • opening conversations about expectations and desired outcomes

An effective rubric needs to provide an accurate assessment of what it is that you want to measure.  So, in addition to using a template, I wanted to discuss some things to consider when you create or modify a rubric.  To start, I want to build on a previous blog post by Phil Gaiser about rubrics.  From his site, is an illustration of a typical rubric:

As you can see, a rubric is broken down into four parts:

  • task description – specifies what is being evaluated
  • dimensions – these list the standards, criteria, or components that you will be evaluating
  • scale – these rank from highest score to lowest.  They can be both numeric values (in terms of point values), or descriptors (excellent, good, average, below average, poor)
  • descriptions of dimensions – these explicitly detail the standards for performance

To create a rubric, consider the following questions:

  1. What are you going to assess? (Task)
  2. What are the characteristics of what you are going to assess? (Dimensions)
  3. What do the characteristics of the highest scoring standards look like? (scale and descriptions)
  4. What do the characteristics of the lowest scoring standards look like? (scale and descriptions)

For example, let’s think about an instructor assessment.  One of the dimensions that I assessed was attendance.  For descriptions, I would use the following:

  • 4 = 100% Attendance, instructor arrived before the start of class each and every time
  • 3 = 100% Attendance, instructor arrived before or at the start of class.
  • 2 = 95% Attendance, instructor arrived at the start of class
  • 1 = Less than 95% attendance and/or instructor was late to class on at least one occasion

Now, you will note that the descriptions reflect a very high standard for instructor attendance.  This was because as soon as an instructor was late to one class, they would receive a written warning – attendance was very important for me, because when an instructor showed up late, it was also a bad example to students.  However, if the rubric had not been given and discussed ahead of time, it could be quite possible that an instructor would think that a 90% attendance would have been acceptable, even though, in doing so, they would score very poorly based on the rubric.

This is another reason why giving rubrics to employees ahead of time can be very useful, because it can prevent misunderstandings.  Here are some more tips to help you:

  • Start with a template to give you an idea of the dimensions that you may want to use.  Draw from several examples to get a feel for what you want to assess.
  • Scale accordingly.  If attendance is not as important as organization, then make sure that organization is given more points than attendance.
  • Make sure your descriptions are measurable:
    • Quantitative descriptions (ie. attended 3 out of 4 meetings) are easier to measure than qualitative (ie. positive attitude) measurements.
    • Details matter
  • Adjust when needed, ask for others to help you.  Especially when trying a rubric for the first time, I find that it may take a few tweaks to improve how things are being measured.
  • It is important to train people accordingly!  Even though two supervisors may use the same rubric, how that rubric is interpreted may be different.  Hence, it is important to “calibrate” yourself to the rubric – and to make sure you are evaluating consistently.  Discussions on what you expect to see can often clarify any ambiguities.
  • Include those that are being evaluated in the developmental process, so that both you and your students or employees are on the same page when it comes to assessment.

For more information, you can go to:

How to rubrics – This document provides a detailed list of questions that can help guide you in the developmental process

Stevens, D. D., & Levi, A. J. (2011). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Stylus Publishing, LLC.

 

Hey, you! Teacher!

hooks quote

Everyone is a teacher.  However, our society has led us to believe that only specific subjects may be taught, and these subjects must be taught in a specific place, by specific people.  Yet, most of the skills we learn about being who we are, being better people, and being good at our jobs are not taught in a classroom.  Yes, formalized education gets our foot in the door for certain things, but our life mentors and teachers, like family, friends, and even strangers, teach us the most about living.

The current research on education challenges this notion that learning is a formalized activity, and that what gets learned should be an amalgamation of decontextualized facts and procedures that for many of us, lack any depth or relevance to our daily living.  In fact, this common notion of learning, and being a “good student” gets challenged in reform-based teaching every day!

For example, in online education, most instructional designers and educators build upon the theory of social constructivism (Mayes & De Freitas, 2004).  This theory says that we construct what we know and understand by actively engaging with others and with our environment (Phillips, 1995).  Now, if you think about how you, personally, have learned things, I bet this makes sense.  How did you learn how to cook?  How to ride a bike?  Or how to pick up a hobby that you love?  More than likely, these things were not learned through a lecture in a classroom.  It may start there, but for most experiences, we learn through doing, talking, trying and failing and trying again.  …and that’s natural!

When we think of learning in this way, then anyone we interact with who has taught us something (and this something could even be a new viewpoint or idea) has become our teacher.  Conversely, when you share your knowledge with others, you are a teacher.

So what are you waiting for?  Go out and teach!

Using rubrics to evaluate students and employees

1This post is in response to a discussion last week at the Second Life Tech Soup Friday meeting.  The speaker, Gentle Heron, talked about employee performance reviews.  She gave tips about how to make them more pleasant; such as providing calendar dates, expectations ahead of time, and opening employee discussions.  I added a comment about how rubrics can help with these assessments…which lead to a suggestion that I do a presentation on rubrics this Friday!

A rubric is an assessment tool that helps score and outline performance expectations.

It got me thinking, ‘I bet most people haven’t been taught to use rubrics effectively!’  When people have an opportunity to rate themselves, and when they know that this rating counts, they take more responsibility to develop an awareness for what they do.  As supervisors and teachers, it is only fair to provide these expectations ahead of time. There is a very different feel when one is being judged, versus when one judges themselves.  When people are given the opportunity to critically self-assess, they become more aware of their job, and reflective and critical of their own progress.  A rubric can not only be an assessment too, but it can be an extremely effective teaching and training tool, too!

When I taught biotechnology to high school students, their grade was based on both my assessment, as well as their assessments.  We both filled out the employee rubric separately, then met together to discuss the scores.  Their final grade was an average between my assessment and theirs.

Here’s some tips:

  • Start with a very clear rubric that outlines all the expectations for employee/student performance.  Sometimes, you may not know what all these expectations are, initially.  If you don’t, this is a GREAT opportunity to work with your employee or student to develop the rubric together.
  • Give this rubric to people ahead of time.  Talk about what the assessment looks like.  For example, ask, “What does a score of 5 look like in terms of attendance?  What about a score of 3?  …and 0?”  Begin these conversations now, so that there are no surprises.  That way, the rubric categories set a standard, and will not be taken as a personal affront.
  • Evaluate mostly formatively and occasionally summatively.  
    • Formative assessments mean that you and the employee/student look at performance periodically, and reflect/revise as you go along.  Think of a chef when they’re cooking an elaborate stew.  When they are constantly tasting the soup, adding spices here and there, adjusting things during the process, they are doing formative assessments.
    • Summative assessments are at the end.  Personally, I do not think that these are as useful, but yet the world (starting with education) has somehow used them as standard.  Summative assessment is like the final taste test for the food contest.  The food is all cooked, finished, and there’s no going back.  This is really tough on an employee/student – and if you think about it, where is the opportunity to learn?
  • Provide spaces between evaluations to reflect, revise, and adjust not only employee/student performance, but the rubric, itself.
    • As a supervisor or teacher, be a “guide on the side,” and mentor – encourage, ask questions, and push, but don’t dictate.  From my experience, people “own” their work and their self assessments when they not only understand what is expected of them, but also that their voices, their input counts.
      • Start by asking the employee/student to justify their scores.  This will give you a good idea of whether their interpretations of the rubrics match your expectations.
    • Assessment and evaluation should be a conversation with clear expectations and understanding.

Here’s a list of places to go for rubric designs and example templates.

iRubric – This site provides starter templates for you to design employee rubrics.  I would start with a generic template, then pull out the job description (or your learning goals), and then customize from there.

Rubistar – This is a great rubric website for teachers.  Similar to iRubric, you can take a pre-made template and customize it to your needs.

Cooper’s Rubric Presentation – Talks more in detail about rubrics, and includes several different types of employee performance rubrics, as well as outlines the steps on how to develop one.

You can also Google “Employee performance rubrics,” “Student rubrics,” “Music rubrics,” etc.  to get a base template to begin.  Then, customize it to your needs.

Do you have questions about rubrics?  Comments?  Please ask away and I will try my best to answer!

Next up on rubrics:  Creating them.

Post Navigation