Data collection doesn't always have to be about creating a survey. In many cases, it can be as simple as finding a structured way to evaluate the things that are happening around you every day. Below are a few ways to break out of the survey bubble:
Portfolio: A collection of artifacts or work that provide evidence of student learning or program improvement.
Strengths:
- Shows progress over time
- Reflective in nature (encourages reflective learning)
- Provides deep examples
- Multidimensional (shows learning in different ways)
- Provides both individual and program-level feedback
- Provides both numbers and descriptive information
Challenges:
- Requires planning ahead (pre-determined outcomes, criteria for meeting outcome, experiences to be included, type of reflection, rating tool)
- Takes time to implement and see progress
- Need trained evaluators
- Need system of collecting portfolios (electronic, hard copy)
- Depending on technology resources, combining aggregate data can take time
Resources needed:
- Do you have outcomes, criteria, learning experience, and reflection prompts prepared?
- Do you need to train evaluators?
- Do you have a system for collecting portfolio materials?
- Do you have time to look through portfolios and analyze evidence?
Observation: A systematic method of collecting data through unobtrusive visual means (e.g., watching people or places) in order to collect information.
Strengths:
- Unobtrusive – does not require participant engagement
- Requires seeing beyond nature perspective
- Often effective with physical plant and watching for student trends
- Useful for gathering initial data to couple with survey or focus group
- Provides both numbers and descriptive information
Challenges:
- Requires planning ahead (e.g., protocols, charts, journals)
- Non-responsive in nature
- Limited in the type of data it can collect
- Need trained observers
- Need system of collecting information
Resources needed:
- Do you have a protocol?
- Do you need to train observers?
- What is your timeline?
Document Analysis: A form of qualitative research in which documents are used to give voice, interpretation and meaning. Any document can be used, common documents may be: application materials, student newspaper or publications, marketing materials, meeting minutes, strategic planning documents, etc.
Strengths:
- Documents are readily available
- Documents are already collected or easily collected
- Low costs
- Documents are a stable data source (they don’t change)
- Can be collected on a quick timeline
Challenges:
- Non-responsive in nature
- Documents are context and language specific
- Documents are often disconnected from their creator
- All documents are written through a lens, need to be aware of lens in order to assess objectivity
- Data analysis takes time
Resources needed:
- How do you gain access to the documents?
- Do you know how to set up a coding system?
One-Minute Assessment: Very short assessments of what a participant is “taking away” from their experience. Should be targeted at a specific learning or program outcome.
Strengths:
- Provides a quick summary of take away from student perspective
- Quickly identifies areas of weakness and strengths for formative assessment
- Can track changes over time (short-term)
- Non-verbal (provides classroom feedback from all students)
- Captures student voice
- Short time commitment
- Provides immediate feedback
Challenges:
- Non-responsive
- Short (so you may lose specifics)
- Sometimes hard to interpret
- Need very specific prompts in order to get “good” data
- Plan logistics ahead of time and leave time during program/course
- May need to be collected over time
Resources needed:
- Do you have a strong prompt?
- Have you reserved time to collect data?
- Do you have a system for collecting data in a non-rushed manner?
Visual Methods: Captures images as a main form of data collection, usually also includes captions or a journal to accompany images. Most often used for photo journals, video projects, and visual art projects.
Strengths:
- More detail and depth to data
- Visual aspect allows for depth in sharing results
- High levels of student investment
- Can use images captured for multiple uses
- Very descriptive in nature
Challenges:
- Beware of threats to alterations of images (especially with technology)
- Usually smaller number of perspectives
- Time for implementation and follow-through
- Analysis takes time
- Resources may be needed in order to capture images
Resources needed:
- How will your participants capture images (resources)?
- What prompt will you use to make sure participants have a clear direction?
- Do you have time to gather and process information in your timeline?
- Have you accounted for time for member checking?
Case Study: A form of qualitative descriptive research, the case study looks intensely at an individual, culture, organization or event/incident.
Strengths:
- More detail and depth to data
- Multiple perspectives are gathered
- Tells a story
- Very descriptive in nature
Challenges:
- Takes significant time to gather information and analyze
- More perspectives = more time
- Narrow purpose as far as sharing data afterward
- Analysis takes time
- Resources may be needed in order to capture data
- Not meant to be generalizable but can be transferrable
Resources needed:
- How will you capture data?
- Do you have a clear understanding what you are profiling and why?
- Do you have time to gather and process information?
- Have you allocated time for member checking?