The Sims 4 Quality Assurance Official Q&A

Question: A really basic question, but what does the QA team do?

SimDevAlex: Great question! At a very simple level we “play the game” and try to find the ways to “break it”. But beyond that we try to be the voice of the players for the devs. It’s not enough to say the game “works” we also try our best to help figure out whether or not it’s fun!

Question: Hello what are you testing/working on at the moment? And what are the plans for future improvements of the game in terms of graphics and performance?

SimDevAlex: So my job specifically involves the stuff ya’ll post in EA Forums Bug Reports. I work with my team to help get to the bottom of what’s causing your issues, and try to get them filed as bugs for our dev team to fix! As for future improvements we have the Memory Boost build available for PC that we are still collecting data on and think that it’ll help in a lot of ways! What would you like to see?

Question: How do you guys decide which bugs to prioritize? i ask because some bugs seem less game breaking than others and they get fixed first, but maybe there is more to it behind the scenes that we don’t see?

SimDevAlex: Prioritizing takes many things into account. We’ve mentioned this before, but we try to priortize “Top issues” (where we consider the votes ya’ll put on them in the Forums) But we also don’t want things to just sit. If we can fix them (safely) we dont want to sit on them! Different bugs have different challenges, some of the less game breaking bugs are simpler/faster to fix than others.

Question: How do we request a QA with someone that can answer bug questions? Seems like something the majority has questions about.

Shawnee (Community Manager): It’s quite complicated to have a team member who can report on the status of every bug as there are different teams and it’s a lot of folks and not a single person tracks the status of every bug by themselves.

Question: That sounds like a fun job, but really time consuming. Do you play with all the packs or just with the base game and the pack you’re testing?

SimDevAlex: I’m very glad someone asked this! (and this may be a 2 parter) But the short answer is “we test with as many combinations as we think can be relevent. Base game and the new expansion is one way, but we also do all packs, and then a variety of combos.

BUT We cant test ALL combos of packs: I’m not great with math, but to show the numbers for why we “can’t test everything”: For TS4, there’s 20 factorial (20! or ~2,432,902,008,176,640,000) possible combinations of just expansion packs a player COULD own in any combination. If we spent 1 second testing each combination it would take 77 BILLION YEARS to validate they all work (for just 1 second). (This doesn’t account for GPs, or SPs, CC + mods.)

This doesn’t account for the variation between Console Types (Sony / Xbox), or what type of PC or Mac you’re using, the screen size, the in-game settings that you’re using yadda yadda yadda. All of this is just “the different variables that can happen before the game even launches”.

There’s practically an infinite number of things that a player can do in their game, while the game is booted, that could trigger a bug. This can range from “this is so silly no one would ever do it” (like clicking your mouse 1 trillion times while holding enter+space bar+caps lock) to what we in QA call “golden path” actions (like entering into the game and playing it “as designed). (This is true for all games and is part of why we love video games!) (If doing this causes a bug let me know please!)

AND THEN Consider that TS4 has a lot of “randomness” that affects things like NPC behavior, weather, events, etc. This randomness provides some tuning flare, which is great for gameplay! But even if we could determine every possible human route, we wouldn’t be able to reliably account for the randomness that’s part of the game. So even if 2 people do the exact same steps, their outcomes could be drastically different!

Question: Do you test each asset individually?

SimDevAlex: Yes! Every asset put into the game has to go through QA approval. If you’re asking “do we catch every bug with every asset” the answer is “We really do try! But we are human and sometimes things are missed!” (But we are fortunate enough to have a passionate fan base who’ll let us know when and how we missed stuff! Which helps us get better over time!)

Question: This might not be something you can speak to Alex, but I’d love to know if you or others in your team ever work on things in the Feedback Forum as well as the Bug Forum.

SimDevAlex: Me specifically…. Yes! While the feedback forums is not officially bugs, we do raise some of these concerns with the dev team. Sometimes a design just didn’t hit the way we thought it might, and feedback helps us recalibrate! Every so often we will pull things out of feedback and into the bug reports if we feel this is the case!

Question: From the maths (sorry, I’m British!), it looks impossible to be able to test for every eventuality. How do you then decide a pack is as ready as it can be for release?

SimDevAlex: You’re crushing the questions I’m double checking you’re not a plant! lol But to answer: we have validation checklists we do.

We spend a lot of time “halo testing” (like testing around features) as well as designing test cases and working with the devs to understand what was changed so we can decide how much time to spend. If Feature A wasn’t touched in the update, we make sure it still works so we can focus on Feature B that had extensive changes in the code or assets.

Question: Hey hey! I am seeing you guys do amazing work and I love that there is this Q&A. I am wondering how you guys take into account feedback and widely requested features. Once again no need to get into the specific topics! I am just wondering how to feedback is processed and if there is any public reaction on feedback.

SimDevAlex: So here’s a fun little diagram i had to get approved from legal to send here lol But effectively, this is the “path” player reports take from YOU to… well you again (but this time fixed) We take your player reports (both in bug reports and feedback), investigate it, get bugs in (or info for the dev) so we can action on it. Sometimes for feedback that investigation is just “more info for ya’ll” and sometimes it results in tweaks!

Question: I am curious though how the testing process goes. Is stuff for instance usually tested with all packs and just the base game? What do you prioritize since you can’t test all?

SimDevAlex: There was a big copy pasta that can basically be summed up with “We analyze risk, and determine what is most likely broken” In QA we have a general “principal” of “bugs cluster” (kinda like roaches in an apartment. There’s never just one!)

Question: When testing new packs do you guys play on different type of computers to see how the game runs on various machines? Like high-end super computers vs potatoes. I’m sure it would be ideal to have the game run great on all kinds of machines, but do you guys ever notice any quality differences based on what you’re playing on? Also, when a new pack goes gold is ready for launch, what is it that typically causes bugs to occur at launch for regular players? Would it be coding related? Computer related? Mod related? I know computer engineering and coding in general can super weird at times so I’m interested in learning what it’s like to deal with things like unexpected headaches after a game has been fully tested and proved to go gold.

SimDevAlex: For the first Q; Yes! It’s common in the industry to test on what we call “Min Spec”. We publish what our minimum specificiations are and that’s what we test on. (And then of course we have good computers too! And everything in between. Different graphics cards, memory, all that tech stuff) We absolutely notice different quality differences, which is why we benchmark a “minspec”. We have standards we try to meet across the board, but we have an “ideal” set up, and a “worst case” set up but we try to maintain quality as best we can. The bugs that we experience at launch can be from a number of different “root causes”. What it boils down to in QA is the old saying “the best playtest is the day your game launches” there’s so many possible variations on what ya’ll do in your game that we simply can’t account for (BUT that doesnt stop us from trying to get better!)

Question: How much say do QAs have on something that can be considered a “design choice” i.e. technically-not-a-bug? Something that could still use some feedback/changes?

SimDevAlex: There is an expectation that we give our feedback for design choices. (And we do!) Our job is to represent the player internally. It helps if we have data that we can bring to the table (like feedback posts in the forums!) but ultimately it’s not our job to dictate what is going in.

Question: At what point in the development process of a new pack do you get involved?

SimDevAlex: The best QA teams are the teams that are involved EARLY and OFTEN. I always tell Dev that the best bug is the “one I never have to write.” (Not because I’m lazy!!) If I can identify a potential pitfall BEFORE the dev even sits down and writes the code, then that’s a win all around. Then I dont have to write the bug, and they dont have to fix the bug, and then I dont have to revalidate that the fix works! Saves a ton of time and clears us up to look for what we call “edge case” testing.

Question: Do you use automated testing tools or is all testing done manually?

SimDevAlex: We have automation, but there’s nothing like getting real human input. Automation can verify something techincally works, but can’t tell you how it FEELS as a player!

Question: Can you tell me more about the more info for y’all? And is it possible to tell more about the communication way? I am not sure how to phrase it well but I have seen some negativity in the community (unfortunately) regarding certain topics. Such as people talking about mermaid/alien skill trees. Or that after the Spa Day refresh there seemed to be a plan to have more but we don’t really have more information on it. I am not sure how to phrase my question properly haha but I guess how do you guys try to make sure that the community feels heard? And that there is follow-up on cases like the refresh/skill tree stuff. Or does the team plan on keeping things a surprise?

SimDevAlex: I’m not keen on ALL the plans, but I know that one of the reasons why I’m here right now is because we are trying to expand this very thing! I’m sure they have more stuff planned in the future, so I reckon “stay tuned” here and everywhere else we post things! (and keep posting in the forums that helps me!)

Question: How does the team decide when to move something to feedback and when its a bugreport?

SimDevAlex: This typically involves an investigation step with the dev team. If we’re not sure we bug it, then when we get the answer we will move it. “better be safe than sorry” when it comes to bug.

Question: Hi SimDevAlex, I did a tiny bit of QA testing at my old job. What is one of the most frustrating parts (for you) when having to test software?

SimDevAlex: Gotta be the randomness. In “non game” software you don’t get this as much, but for games, trying to find repro steps that involve randomness is the most frustrating thing. Or timing specific bugs.

Question: How do you plan your testing new content? Do you have certain scenarios you play through every time and then others that are designed for the specific thing you are testing? Just wondering how you narrow the millions of things players can do down to what you actually focus on for testing.

SimDevAlex: We write out and design different types of test cases. From validation checks (“does it work?” ) to playerbased testing (“is it fun?”) to edge case testing (“What happens if….”) The concepts are the same across all games, but the routes are different depending on the content. (Testing Horse Ranch is different than Cats and Dogs despite them both being heavy on 4 legged friends)

The idea is to figure out “what is the riskiest thing” “what is the stuff most likely to break” and then go from there. It’s really an art form and goes beyond just “booting the game and farting around” (But tbf we do that too sometimes lol)

Question: To me Adventure Awaits seems to be coming from player input in a past feedback form. I am loving those efforts yeah. Personally I feel like updates on majorly requested stuff could help and also with expectations. I.e. with pack refreshes Spa Day raised some expectations on if it will be done in the future. I believe that a mention of: We have plans in the future OR at the moment we don’t have plans can manage expectations. And maybe also take away some negativity people show when speculating. Imo a we don’t have plans at the moment is always an answer too.

SimDevAlex: Adventure Awaits is a good example of us taking ya’ll’s feedback seriously. We didn’t intend on having the new curtain be modular, but after there was public feedback we decided that it was something we should change. And now it’s something that’ll actively discussed in the future. (Plus the change was pretty low risk so it was quick to get in and test!)

Question: There are so many amazing things created. And sure there will be bugs, some stuff will be missing. But I feel like there is a big bias of focus on the missing parts than on the amazing parts that exist. And the team is putting much effort into improving communication efforts which helps the community voice things for fixing. I.e. we can look at 1 bug unfixed or see the 100 fixed bugs.

SimDevAlex: We’re also looking for ways to better improve the loop between the players , us in QA , and the Dev team. TS4 isnt the first game I’ve worked on and I can tell you our game is pretty special in both the tools we use to collect player reports (EA Forums) and the fact that we have a community of Simmers who are so passionate about all aspects of the game that they come to us with their frustrations.

From my POV, it’s amazing how much ya’ll care and that you are willing to engage, and we are constantly striving to better our processes to meet a level of quality that matches that!

Question: A general QA question: when testing, do you guys just play the game as if you were a regular customer playing on launch day and take note of bugs as you go? And how long do you guys typically play the game when testing?

SimDevAlex: We do do this! We call it “ad-hoc” testing. Generally it’s best practice for all QA (not just TS4) to keep a 1 :1 ratio of “technical testing” and ad-hoc. (so for every hour we spend doing check lists, we do an hour of gameplay.) This keeps our minds fresh (because checklists are tedious) and also lets us explore and get better at the game and better understand it through different lenses!

Question: Does looking out for balancing fall onto the QA team? Such as the costs of items and how they fit into the already laid out game “economy”, and for the balancing of game mechanics such as those of luck, romance, skill gains, etc. And whether or not new game mechanics have unintended chain reaction effects on other parts of the game in terms of proper gameplay balance.

SimDevAlex: Quick answer yes, kinda. We can give feedback on how it affects gameplay but ultimately it’s not our call.