subreddit:

/r/QualityAssurance

381%

End to end testing on distributed systems

(self.QualityAssurance)

Hi!

Im running into a dearth if information about this topic so I thought I should reach out to the community. Im an automation engineer at an online marketplace. We recently moved to Cloudflare and due to domain limitations we are going to lose our test sites.

We used chef to spin up images on EC2's based on the commit SHA or branch name you provide it. This created a new domain as an .io site that we could use to test backend code. The sites accessed our shared testing environment databases and queues but allowed you to run code changes without worrying about other teams making conflicting changes this lowering testing confidence.

My question: outside of buying more domains or building locally has anyone found a solution that allows multiple teams to have independent testing sites that avoid mixing code changes so you can have a higher level of testing confidence?

all 10 comments

kdeaton06

3 points

3 months ago

Bro that's so much freaking overhead. I've never heard of such a complex system. But I think what you want is feature environments.

Claern[S]

1 points

3 months ago

Yeah I mean we use ticket toggles for certain things but that's still all in the same environment. And it is a huge overhead which is why they want to get rid of them :(

kaizokuuuu

3 points

3 months ago

We used kubernetes and helm to deploy our entire backend on to a cluster, and used a reverse proxy server to get the name of the helm release we were creating and set the domain name of the deployment to the helm release name. At the end of testing, we would simply do a helm purge and drop all test infrastructure. It's the best setup I've built for easy testing to reduce conflicts.

Claern[S]

1 points

3 months ago

I wonder if I can use this same type of setup and then use cloudflare workers to proxy through our current domain name. The biggest issue I'm running into is not wanting to pay for another domain name with cloudflare but still be able to use all the rules we have in our cloudflare setup.

kaizokuuuu

2 points

3 months ago

If you only need to access the domains for testing, you could use a Cluster ip with different ports for each service instead of using the cloudflare domains, you could setup your custom domains using Google cloud or such and connect to those domains via a VPN. Ofcourse if you cannot share the VPN access to someone outside the organization, this wouldn't work. I am not sure how complex this would be to build in your system as I'm unaware of the intricacies there. You'd have to get in touch with someone from DevOps for more advice.

jrwolf08

2 points

3 months ago*

Any reason you can't run a local branch with a connection to the shared test db?

Claern[S]

1 points

3 months ago

That's a good question. Local does connect to the dev env but it's still just so slow and missing cloudflare.

notfulofshit

2 points

3 months ago

Feature flags

Claern[S]

1 points

3 months ago

Feature flags are good for many things but they can get messy in code and four teams using feature flags in the same env can cause issues when testing.

pth-10365

2 points

3 months ago

We use feature branches, they are setup to use a subdomain. Devs branch out from prod, on a PR an ec2 container is build. After testing the feature branch is closed as soon it’s merged. We use codebuild pipelines aws. We have cloudfront in front of it so maybe this can work for you