How to properly think about automated strategy?

Hello everyone.
I am a manual QA and in my current company we are beginning to work on creating e2e automated tests using Playwright with TypeScript.
What environment should I target ? It makes sense to me to develop automated test suite using the data of “test”/ “stage” environment since the goal is to reduce the time of manual regression/smoke testing. I understand that ideally I should be able to make those tests in a way that they can be used for any environment but I have doubts this can be done with our application.
I will appreciate if you could share your thoughts / experiences.
Thank you and have a great day!

My experience may not be typical because I’m the only tester in the division, so my time for automation is somewhat… limited.

That said, I’d suggest starting with the test environment, because that’s where a breaking change will show up first (most of the time).

What I’m doing now that I have a mostly stable suite of automated tests is running them on the QA environment and the Staging environment. I had them on QA only to start with, and only set up the Staging environment once I had the tests stable and as comprehensive as they’re likely to get (for a variety of reasons our software has a lot of modules that aren’t amenable to automated regression).

Once I have them stable on both environments (which is taking some tweaking and some adjustment), I’ll set them up on the production environment using dedicated test organizations (we support a multi-tenant web application so we have a set of test organizations). The production tests will be happening off-hours so they don’t interfere with customer traffic.

The way I do this is to put the environment information such as base urls into a test settings file. I have the ability to override this in the CI/CD tool we use, but you can use different flags in your automation to manage environment-specific information.

I don’t know the specifics of using Playwright, but in my view going with CI/CD to automate your test runs and starting with the environment that’s closest to where the changes are happening is the best way to catch potential regressions early.

Viola , since you say you are a manual tester, starting to use an automation toolstack? Right, I’ll frame things there.

  1. Were just talking about the differences between Staging and Production with mates, when it comes to kaptchas. This is a common journey blocker for automated tests and we disable these in staging for that reason. This and loads of other “common-knowledge” aspects of automation become long-term friction to your automation efforts. Environment security changes can block running the same test in all environments, plan coverage for gaps ahead.
  2. So when you use a off-the-shelf toolstack, another friction area is functional block or code reuse in an E2E, where some blocks or pages get hit very often, and become bottlenecks for the tool whenever they break. As your number of tests grow they will have a lot of dependencies on just some key shared steps. Try to prevent this as far as possible so that one page in your app undergoing a legitimate change does not fail every single test in your suite, manual testers never hit this, but automation is very susceptible, so try spread that risk.
  3. There are many ways to do so that involve using back-end to get you into specific states to leapfrog potential bottlenecks - this is going to require coding skills…or at least being very good friends with a coder.
    You will be creating “software”, software architecture and designing that code for maintainability are skills you will need to learn. I have worked with some smart manual testers in the past, and they all had to work hard to think in patterns and organize and keep tidy their manual test cases and rig setup and environments. The same tidiness applies to code, even more so. It’s a journey, be prepared to start over from scratch, in fact assume that you will be learning a lot, so try to make it cheap to wipe and start over.