Herding cats online

From Driscollwiki

Williams, D. & Xiong, L. (2009) Herding cats online: Real studies of virtual communities. In Hargittai, E. (Ed.) Research Confidential, pp. 122-140. Ann Arbor: University of Michigan Press.

Challenges of studying "the anonymous online gamer" (122)

• Working with companies involves NDA (Non-disclosure agreements)
  • "A legal document that limits what you can say to people who are not associated with the project or company" (123)

Contents 1 Groundwork, planning 1.1 Data collection 1.2 Working w IRB 1.3 Corporate relations 1.3.1 Axioms 1.4 Political reality check 2 Execution 2.1 Choosing and motivating respondents 2.2 Experimental design 2.3 Recruiting 2.4 Retaining 2.5 Debriefing, feedback 3 Note on surveys

Groundwork, planning

Data collection

• Sometimes we can work from existing datasets
• More often than not, we must collect it ourselves (123)

Working w IRB

Explaining unusual research (124)

• Minimize risks
• Informed consent

Be thorough

• Know all the risks ahead of time
• Know the likelihood of children being present

How to get this information? (125)

• Interviewing
• Participant observation

Corporate relations

• Positive: "unobtrusive behavioral data" (Webb et al. 1966)
  • Especially possible when you have access to companies' internal data (125)

Axioms

• Understand the company's perspective (125)
  • e.g. Microsoft lawyers were not loving the constructs in an effects-oriented study (126)
• Networking matters (127)
  • Face-to-face, person connection of mutual interest
  • For the game researcher: GDC

"[Motivating factors for the company?] Build and demonstrate advanced, rigorous, and accessible tools that both help them understand who quits the game and why, and help us test our hypotheses" (129).

Political reality check

• Patience, persistence (129)
• "Stick to the science and studiously avoid the personal" (130)

Execution

Choosing and motivating respondents

• Challenge to choosing a "truly random" sample (131)
  • In WoW, they developed categories of players and ensured that they had equal representation of all categories (131)

Experimental design

• Lab
  • Positive: very controlled
  • Negative: Hawthorne effect, they perform unnaturally in a lab environ

Recruiting

• Money isn't the only motivator
  • 10k respondents to a survey with promise of a special item (133)
• Shotgun blast requests on fora may be understood as spam (134-5)
• Researchers may find they alienate less subjects by approaching people within the bounds of the site
  • e.g. Contacting players from in-game (135)

Retaining

• Contact with leaders enabled effective snowball sampling
  • And a trusted contact in the field (136)
• Planning ahead for "volatility", attrition, possibility of high dropout rates (136)

Debriefing, feedback

• Post-test survey
• E-mail availability

Note on surveys

• Interpretive challenge
• Pre- and post- surveys may not be sufficient
• Helpful to have periodic contact (weekly/monthly) (137)
• Technical errors likely: file formats, bugs
  • Always run a pilot (137)
• Effective survey design depends on knowledge of social context
  • Always a participant-observation step (138)