#Survey design

[ao3demographicssurvey2024]

Hey everyone!

As we are working on question design, the AO3 Demographics Survey 2023 will be running a series of pilot surveys. These are to seek feedback about which questions should be included, the best phrasing of questions, and similar design concerns.

We are now asking for feedback on questions from the "AO3 Usage" section of the survey. This section deals with how participants use the site Archive Of Our Own and their preferences for different types of fic. If you are willing to give feedback, we estimate the survey will take about 15 minutes. No data collected as part of this survey will be included in the final project; it is for survey design purposes only.

This pilot survey will run for about a week and we hope to get around 75-150 responses. Signal boosts are appreciated. Thank you!

[gendercensus]

How many pronouns should be in the pronoun checkbox list?

So, I'm working on generating the pronouns checkbox list for next year using the multiplier system.

The following options that are not actually pronoun sets have to be on the list:

I want people to frequently change the pronouns they use for me

My pronouns vary depending on specific conditions

Any

Avoid pronouns / use name as pronoun

A pronoun set not listed here

Questioning or unknown

For comparison, the identity checkbox list is 20 long, in two columns (10 per column), which seems to work. That looks like this:

This year's pronoun list was 15 long. It can't be over two columns because the options are a bit too long. On the same screen, 15 checkboxes look like this:

(Bear in mind the "preview" black bar at the top won't be there for everyone else!)

What do you think is a good length for a pronoun list in one column? Think: Plenty to choose from, but not so many that you have to scroll too much and can't easily find what you're looking for.

Feedback also tangentially welcome on the length of the list of pronouns or the list of identity terms in this year's survey!

[numbers-have-feelings-too]

Sometimes, people mess with the census as a protest because the government is denying them human rights, sure. But an unavoidable problem in survey design is that people will also sometimes tell you complete untruths just because they think it’s funny.

Census Jedis are your classic example. In 2001, censuses in the UK, Australia and New Zealand asked about religion for the first time. 2001 was also the early days of widespread internet, and so a meme began to circulate in chain emails.

People were urged to put ‘Jedi’ as their religion on the census, partly for fun and also (they were told, though this isn’t actually true) because it would force their government to recognise it as an official religion.

People did. In England and Wales, 390,000 people did, outnumbering several minority religions.

And why? Some people framed it as a protest against asking about religion on the census, or against the census in general. And sure, there’s an anti-establishment sort of feeling behind it.

But for my money, people mostly did it because it amused them. Because surveys are boring, and people get very serious about the census in particular. Phrases like ‘civic duty’ get thrown around. And there’s a form with all sorts of serious boxes to tick, but also a free space to write whatever you want, and, well, it would be funny, wouldn’t it? It just would.

There are consequences to these sorts of shenanigans. Questions were asked about the quality of census data. Atheist and Humanist organisations begged non-religious people to stop doing it, so as not to artificially inflate the count of religious people. Sociologists have written papers.

But all of that just makes it funnier, doesn’t it?

Look, I’m not encouraging doing this. Nobody who works with this sort of data in real life can afford to – we’re all terrified that the rest of you will decide something else is funny next, and muck up a carefully-calibrated data series we’ve been using for years.

I’m just reminding everyone to beware. There’s no power on Earth that will keep people from being silly, including within your datasets. Sometimes you just have to work with it.

[queercripintersex]

I really want people who design surveys and forms to fill in where sex is a question to get it in their heads that intersex is not a third sex.

I’m so tired of encountering forms that are like: “Your sex: Male Female Intersex”

But then will only let you pick one option. I’m an intersex woman who happens to produce a bunch of testosterone.

I, like most intersex people I know, do match up to one of male or female. Some intersex people do not, so there needs to be a way to have neither male nor female be available.

What I want is to be able to check all relevant options - female *and* intersex.

Additionally: I want even more than that is to not have to deal with the question at all unless strictly necessary. It makes sense at a doctor’s office to fill it in but it drives me batty that I have to deal with questions of sex (which are frequently conflated with gender) just to do basic things like sign up to a website to buy a widget.

[writers-worst-nightmare]

Survey

If you guys can take this survey and / or pass it around to your friends that would be such a big help! I am trying to prove to my book agent that there is an audience for the type of book I am writing.

Do not feel obligated by any means to take this, but if you want to then please do :)

[marketxcel]

5 Methods of Data Collection for Quantitative Research

Discover five powerful techniques for gathering quantitative data in research, essential for uncovering trends, patterns, and correlations. Explore proven methodologies that empower researchers to collect and analyze data effectively.

[designaday]

Cumbersome

I went through an interesting thought experiment today. When you are designing a survey, it's important to think about whether you are providing a positive or negative statement, how that corresponds to the rating scale, and whether or not it is consistent with other statements in the survey. The System Usability Scale (SUS) survey, for example, alternates between positive and negative statements. It does so on purpose to avoid response biases. The eighth statement in the SUS survey is, "I found the system very cumbersome to use."

This is a negative statement, and the user must answer by indicating how strongly they agree with the statement on a scale of 1 to 5, 5 being the strongest agreement. What if you wanted to make all of the statements positive?

We know exactly what it means for a system to be cumbersome: it performs slowly, it takes too many steps to complete what should be a simple task, you have to do the same thing repeatedly, and it's hard to remember where to find things. What is the opposite of cumbersome?

A thesaurus isn't particularly helpful. Some antonyms listed for cumbersome are "aiding, convenient, easy, fun, helpful, undburdensome." Some of the words can fit into the statement and make sense, but they don't quite capture everything that the word "cumbersome" does. "Convenient" is probably the closest.

The best phrase I could come up with is, "I found the system efficient to use."

[ao3demographicssurvey2024]

Hey everyone!

This is the next in our series of pilot surveys for the AO3 Demographics Survey 2023 - an independent research project about the demographics and behaviours of AO3 users, which is planning to launch later this year. (Please note that we are in no way officially endorsed by or affiliated with AO3!)

The purpose of our pilot surveys is to seek feedback about which questions should be included, the best phrasing of questions, and similar design concerns. This survey focuses on questions from the "Fandom Behaviour" section of the survey, which is about how people participate in fandom beyond their use of AO3 itself.

If you are willing to give feedback, we estimate the survey will take about 15 minutes. No data collected as part of this survey will be included in the final project; it is for survey design purposes only.

This pilot survey will run for about a week, or until we have received 500 responses, whichever happens first. Signal boosts are appreciated. Thank you!

[theperfectquestion]

Welcome to our friendly Discord server!

To verify, just complete this short survey:

1. What are your skills?

🌱 gardening

🍏 horticulture

🍇 viticulture

🌐 other

2. Where are you from?

🇳🇬 Nigeria

🇪🇹 Ethiopia

🇪🇬 Egypt

🇨🇩 DR Congo

🇹🇿 Tanzania

🇿🇦 South Africa

🇰🇪 Kenya

🇺🇬 Uganda

🇸🇸 Sudan

🇩🇿 Algeria

🇲🇦 Morocco

🇪🇺 Europe

🇺🇲 New York

🇮🇳 China / Asia

🌐 other

[a-ux-person]

An old example but a good point about surveys.

NPS (Net Promoter Score) surveys are a pretty classic metric for most companies, but how often do they provide valuable feedback from consumers?

It's a nice number to have, and I won't argue that it should be tracked, but it the NPS needs more context to be valuable, and the above is a good example of that.

The user is caught up in the language of the question (literally recommending a OS to someone), and isn't answering the the question the company is actually asking: How do you feel about Windows 10 and explain your response.

This one is cluttering up the NPS for Windows 10, when in reality the user may enjoy the OS. We'll never know for sure.

A good lesson: Nitpick your survey language and always test the survey!

[queercripintersex]

Today in things I have learnt: when making a tumblr poll for intersex people, present multiple options for the non-intersex people to vote because they are so very numerous and so very curious and will completely dominate the results of your poll to a point where you can't see much of anything else.

Made a poll earlier today on when one’s intersex variation became apparent. Within an hour results were something like 30% puberty, 20% while questioning gender, 5% baby, 5% other care. Just returned to the site 13 hours later and now we’ve got this:

alt text: a poll asking “when did your intersex variation become apparent?” where 78% of respondents are not intersex and so all the other options that are not puberty have been reduced to 1% or 2%, and you have no decimal points or total votes per option to try and figure out if a given 1% is bigger than another 1%. It is followed by a gif of a man, exasperated, bringing his hand to his face:

[fascinatedfinch]

Day 11: Planetary Surveys (from MB's point of view)

[camera-seventeen]

2024 carat survey results! (check the twitter thread for more results ^^)

[queercripintersex]

Also: most intersex people don't consider their gender to be intersex specifically. We have intersex-specific genders like intergender, ipsogender, ultergender, etc, and many intersex people identify with genders that are not intersex-specific.

Don't make intersex people pick between reporting their gender and reporting their intersex status.

hey guys reminder that there are more gender identities than just cis and trans . if you're making a poll that depends on how you identify gender wise include at the absolute bare minimum intersex and nonbinary people . not everyone is cis or trans .

[mayakern]

what if you did this poll telling me which color variants you like for shirts 🤭🤭

what if then i made some of those shirts in the future 😋🤪

that would be SO silly i think