I fell in love with Experience Sampling Methodology (ESM) when I was a first-year doctoral student at the University of Akron. I was struck by how the method could allow me to understand how day-to-day events – both good and bad – affect employees’ emotions, motivation, and well-being. That love has remained steadfast, as I’ve continued to see the method as a valuable tool to truly understand the “lived experiences” of employees – especially groups who need our support now more than ever (e.g., working mothers who are breastfeeding; job seekers during COVID-19).
Although I love the method, I’ll admit that it has challenges during the peer review process. Sometimes reviewers “don’t get it,” with “it” being why the study was within-person in the first place. Other times, there are concerns about the fact that experience sampling studies, by default, tend to rely on self-reports (which truly is at the core of why the method was developed – we wanted to know why individuals’ perceptions of their own experiences at work and home were fluctuating). For me, these critiques have popped up regardless of topic or journal outlet, which always was puzzling as I tried to understand how I could better explain the methods I loved.
Addressing this puzzle ultimately manifested as a paper I’m really proud of at Organizational Research Methods (Gabriel, Podsakoff, Beal, Scott, Sonnentag, Trougakos, & Butts, 2019). To give you the “behind the scenes,” at the Academy of Management Conference in 2016, we (myself and Nathan Podsakoff) held a panel session with Dan Beal, Paul Bliese, Brent Scott, Sabine Sonnentag, and John Trougakos on experience sampling methods. We covered the promises and pitfalls of the method, having all had similar difficulties with reviewers about why experience sampling studies had novelty and value in the field. From this panel, a subset of us, later joined by Marcus Butts, began to dream up a paper – one tackling common issues with the method and our collective expertise for how to address these challenges and make experience sampling studies rigorous and valuable. The paper (Gabriel et al., 2019) is one of the coolest I may ever write – it’s in a Q&A format that lets us tackle some of the big issues of experience sampling methods and weigh in on how we want authors and reviewers to use the method. I also love it because it’s accessible for people at all levels of familiarity with experience sampling methods. We wrote it hoping that both newcomers to the method and long-time users would find something of value that would help them not only design experience sampling studies, but write them up in ways that reviewers and editors alike understand the design and analytic decisions.
It’s impossible to dive into each question in a single blog post! But, if you are puzzling over any of the following 10 questions surrounding ESM, then this paper is for you:
1. How can we use ESM studies in a way to build new organizational theories that are within-person in nature and/or provide evidence that challenges existing theories?
2. What are isomorphism and homology, and what role do they play in ESM studies? What is the value of ESM studies that test isomorphism and homology?
3. ESM studies often focus on maximizing intraindividual (Level 1) observations, but is there a certain number of participants needed to ensure that the Level 2 sample size is sufficient? How many ESM scholars include power analyses in their work? (Spoiler alert: for the latter question, it’s very few!)
4. The frequent, repeated in situ measurements required in ESM studies place a substantial burden on participants. In response, researchers are often concerned with how to incentivize participants to complete multiple surveys in a specified time period. What incentives are employed in ESM studies, and which are the most effective?
5. What techniques should researchers (not) use to provide evidence for the psychometric properties of within-person measures?
6. A challenge in designing ESM studies is balancing the burden placed on the participants and retaining the psychometric properties of the measures being included. How should researchers modify or adapt scales to be applicable in ESM contexts? What evidence can researchers provide for the psychometric properties of short-form scales? Should researchers use single-item measures of constructs in ESM research?
7. If, when, and/or how should a researcher go about testing data for possible trends, and what trends should be tested (e.g., linear/quadratic trends, between-day and within-week trends, within-day trends)? Relatedly, how does one’s research question shape the answers to these questions, and how can it be used to improve the design of the study?
8. What sources of common method biases are typical ESM studies subject to? Does centering scores on Level 1 variables affect these biases, and if so, how? Are other remedies available, and how effective are they at addressing sources of potential common method biases?
9. Are there any consensus recommendations for if, how, and when data from subjects starting the study on different days should be treated?
10. ESM scholars are using a “standard” approach to collecting intraindividual data (e.g., collecting three self-report surveys daily for 10 workdays). However, scholars are shifting toward alternative data sources (e.g., other-reported measures of dependent variables). What are the considerations in collecting data from other sources? How can they be used beyond having another source of data for an outcome variable?
Phew – that’s a lot! And as you can see, there are so many decisions that need to be thought through when it comes to designing experience sampling studies. I hope that by working through these 10 questions with your research team and following the advice we detail in our article, you’ll be able to fend off major reviewer critiques and design within-person studies that truly help us unpack the lived experiences of employees.
Article: Gabriel, A. S., Podsakoff, N. P., Beal, D. J., Scott, B. A., Sonnentag, S., Trougakos, J. P., & Butts, M. M. (2019). Experience sampling methods: A discussion of critical trends and considerations for scholarly advancement. Organizational Research Methods, 22(4), 969–1006. https://doi.org/10.1177/1094428118802626
