One D&D Survey Selection Bias

I stumbled on this video that was posted a couple of weeks ago that I think makes some interesting points. As a stats major, I really appreciated him highlighting how selection bias can skew survey results.

9 Likes

I caught that one. This certainly isn’t a new thing for D&D listening to its most vocal few, but it’s getting well covered this time.

2 Likes

This issue regarding D&D has been discussed many times previously.

'In all my research into TSR’s business, across all the ledgers, notebooks, computer files, and other sources of data, there was one thing I never found - one gaping hole in the mass of data we had available.

No customer profiling information. No feedback. No surveys. No “voice of the customer”. TSR, it seems, knew nothing about the people who kept it alive. The management of the company made decisions based on instinct and gut feelings; not data. They didn’t know how to listen - as an institution, listening to customers was considered something that other companies had to do - TSR lead, everyone else followed.’

In the same thread regarding the D&D Next playtest & surveys :

‘But it’s pretty blatantly obvious that the polls and surveys they put out were NOT designed by someone trying to collect unbiased data. Leading questions and selective answers abounded. I distinctly remember at least one poll during the playtest where there was no answer that was actually opposed; you could choose enthusiastic approval, measured approval, partial approval, or being noncommittal. Plenty of others, the “yes” answers were highly enthusiastic while the “no” answers were not. Yet these were the data points used to shape the design of the game.’

‘I said the playtest was “riddled with problems,” and that you can see that it was by looking at the final product–lots of good ideas mixed with plenty of really rough ones.’

2 Likes

Incredibly valid though. Corporate memory can be short term and generational. The feedback opportunity is great, and is good marketing fodder - but we have to hope that they are, behind the scenes, doing surveys of other segments.

2 Likes

One would hope. But based on my (very limited) experience with WotC folks, it seems a rather slim hope.

Neither TSR or Lizards has been particularly interested in retaining old gamers. Apparently we don’t have disposable income of something.

3 Likes

If one listens to the designers currently with Wizbro, the word that they repeatedly mention is, “story, story, story!” just as Jeremy Crawford did in that above video clip. Here’s another example, https://twitter.com/jeremyecrawford/status/1485803941665280002 .

This is, in my personal opinion, one of the main divides. Currently Wizbro is focusing on selling predetermined stories while gamers who are interested interested in pre-1986 style A/D&D prefer locations that can be used for emergent style gameplay to develop their own plots. We are not going to get this old-school style material from Wizbro & their current designers. I guess it’s because they feel that we cannot be as easily monetized…

1 Like

Give us what we want, not want we are told we want, yes we are. The sheer amount of gaming material in this house speaks to that.

2 Likes

The line you quoted is specifically in reference to this : Wizards of the Coast President Believes Dungeons & Dragons Is "Really Under-Monetized" .
We are less likely to endure being “nickeled-&-dimed” by Wizbro.

1 Like

TL;DR - No Real Merch! Wahtzee is missing real merch opportunities in favor of going after the toxic micro-pick-pocket model of the mobile market.
They are right that the IP is woefully under-utilized. Imagine if they focused on a limited run series as good as the Witcher, but set in Waterdeep. Or if they stole devs from Bethesda to make a massive game to compete with the Elder Scrolls. Those things, in turn, could be monetized to death. If they did a show as good as Game of Thrones, they’d have 15 Funko Pops instead of 3.
FYI, my day-job is merch, so their long standing history of watching other companies eat their lunch has frustrated me for 2 decades now.
As an experiment, imagine you are a non-gamer looking to get a gift for a young DM who won’t stop talking about their new favorite hobby. You go to what you think is the website for the game, and try to buy them a shirt, or something. Click on this link and look for branded merch.

2 Likes

Yes, that is a gaping hole for them. I believe a major part of this issue is that they exclusively rely on external social media, which I think was extremely shortsighted. Recall what they said when they nuked their forums? https://www.quora.com/Why-did-Wizards-of-the-Coast-close-the-D-D-forums

“Choosing to retire a former foundation of our community was not an easy decision, but we feel that we must adjust our communications structure to reflect where conversations about Wizards of the Coast games are taking place.Social media has changed significantly over the last ten years, and discussions about games aren’t exclusive to company-hosted forums. The majority of community conversation takes place on third-party websites (such as Reddit, Twitter, Facebook, Tumblr, and many other fantastic community-run websites), and it is up to us to evolve alongside our players.We encourage past and current users to retrieve any information you want to retain from the Community Forums for both Dungeons & Dragons and Magic: The Gathering. The shutdown will occur on October 29, 2015, at 10:00 a.m. PT. We want to provide enough time for our forum members to move their content, and we recognize that given our forum’s vibrant user base and extensive history, this may take time. Any information still on the forums on the cut-off date will be deleted.Thank you to all of our past and current forum users. You helped build our community into what it is now, and we look forward to continuing to interact with you on our many active social platforms.”

“I could hop onto all the forums having this discussion or I can say it here and let it disseminate. I’m choosing the former. Moving away from running our own forums doesn’t mean we think longer conversations or fan sites/forums aren’t good or necessary. They are both good & necessary. From what I’m seeing, they flourish and you enjoy them more when they are run/managed by fans. DnD & RPGs in general are all about story telling & talking with friends. It makes sense that we want to share those stories. So, it’s vital that we have places to share those experiences & stories, like forums & fansites. But it’s not vital that #dnd run those.Closing our forums does not in any way lessen our interactions. We’ll still be talking & lurking in your social media & fan sites. And the idea that forums are going away because dnd &/or magic are doing poorly - that’s ludicrous! :stuck_out_tongue: Both are doing very, very well.We’ll still be talking with you here, and elsewhere. Enjoy your new forum homes and don’t forget to migrate your treasured content!On the topic of losing forum content - it’s tough, I agree. Once we knew we were going to close the forums, we also knew we weren’t going to maintain the community site indefinitely, so we opted to pull the band-aid off quickly rather than let it linger.”

This is an absolutely ridiculous non-strategy. Literally choosing to rely upon whatever random buzz might happen somewhere, somehow. Bizarre.

1 Like

So, I know nothing about stats and didn’t say anything because I assume they knew more than me. But the moment I heard about one dnd, I immediately thought “I’m super hardcore about dnd, but even I dont know much about the new system and the feedback they’re looking for.” I hope they’re finding a way for the casual gamers to be heard…