Empower your Quality Assurance process with deal pipeline software for Quality Assurance

How to create outlook signature

hello everyone to this little video bertha quarry qi pipeline my name is simon i'm the QA engineer of quarry and let's talk first of all what's curry curry is a story driven story driven narrative cannot in a combat free adventure puzzle game which is heavily inspired by Norse Norse mythology and that's a university project basically so we have been working on this for an entire year we are 30 people in a multidisciplinary team I'm part of the QA team we use Professor as our source control tool and therefore we are trunk based and the main engine we're using is Unreal Engine Unreal Engine 4 and this enables us to deliver this project at the end so we are currently in the release state early access on Steam hmm let's talk about the stages so in our QA pipeline we have the following stage every time and developer creates something it needs to be reviewed it needs to be reviewed by first other people and at in the moment it you create a review wire perforce aka swarm it triggers automatically a built which will build the entire entire review change list and then it will test it it will run the basic essential tests for our gameplay so not all tests this is very important because all tests will take way too long because we test all functionalities like all the quests everything and that would take way too long and we don't have time to do that that's why we have the second step which is we run a full test analysis three times a day in the morning before we work before we start working to catch anything if anyone felt the need to commit anything after work then at lunchtime in order to check if anything broke before lunchtime and then after work at 5:00 in order to check if anything broke during the day and we just run all the tests all the extensive test this test unit takes about an hour to run and then we do also twice a day a development deployment so the first deployment happens in the morning like actually lunchtime technically speaking lunchtime so it happens at lunch time to cheat see if the bill completely can be just deployed the only differences we don't deploy at the end and at the end at 5:00 after the tests have been run we do also a diploma and see if it can be can be deployed and then we do an actual deploy on a daily basis at the same time every day everyone knows when we deploy so we know exactly that this is the time we deploy and before we actually deploy we do the full thing we run the builds we nee tries all blueprints we're gonna test everything extensively and then if that's all been done we push to steam and so let's talk about the tests a little bit we have two types of tests we have functionality test which tests the pure functionality of everything for example does the player get damaged if something happens to to the player or do certain aspects of the game work like in our case do you drain your blessing we have also extensively functionality test effects for box we used to have four we encountered during the development process and those bugs are going to be tested as well if they come back or better not and the second part is we have quest tests they test basically all the quests do they work and if they're not work we're going to fail we're utilizing the Unreal Engine for automation testing to which you can see in this little image here and this allows us to run the tests before people commit so they can see hey that's my change decision actually break something and we also run them on the server which is very important so we can you can run this also via a command line and we have an extensive library for this and that's where we coming next to is our built environment so we're using Jenkins as our continuous integration environment and we're running it with two nodes one is purely dedicated for testing and the other one is purely dedicated for building and deployment this is varying do it the day it's depending on the loads like if one of the nodes is not busy then it takes ant jobs from the other one but if this one is busy it doesn't so like it's prioritized which node takes what ingly to the type of the job in order to make this happen I need you to modify a little bit jenkins moved my own plugin or actually with existing plugins together in composition and then the second part is we have an extensive pipeline script library because all our jobs run with pipeline script like pipeline scripts and we have an extensive spy pipeline script library which supports Unreal Engine 4 and swum in montes integration so we can build we can cook we can pack we can run tests we can commit we can update swarm reviews we can give comments as well we just can't we can download up boards we can create new versions and mounters we can update box we can remove bugs we can add bugs all those kind of things or I can happen automatically if something goes wrong or something triggers it this is very helpful and helps us at the end of the cycle to do less manual labor now we have heard basically about the cons and the cost of the talk and you possibly wonder what are the downsides so let's talk about the problems so the problems it takes time so every review you have need to be first manually reviewed while the build server is validating it by testing and building so this might take time and depending on how many tests you run and depending on what you run and how good to service this might take time so our review time takes about 15 to 20 minutes per review so you can do simple math and you know how many we can do per hour this might be a problem this is especially a problem if something breaks along the line but I'm coming to this later the second problem is with this whole set up because we are trunk based so if something breaks a Kay for example the free tests we are running during the day to validate that nothing actually break the trunk so the master we need to stop nobody is allowed to commit anything till we have fixed this bug this can create a little bit of a traffic jam of commits so and the next thing is with this we need to guarantee that tests working all tests need to work and what we have discovered or experienced in the last month while development on is that when people implement new features or change P features they sometimes forget to update tests which causes that in that technically speaking everything is fine but the tests fail because the test expects that the the old setting so and that's a policy problem there's needs to be clear at the beginning that everyone should run the test locally as well so this can be done solved with that to validate hey you forgot to cheat fix the tests when you do this business and that can be a problem another problem is if one of the tests fails this might be not always to all members of your team clear that it's actually a problem in the test where it's a problem in in the build so if for example a test face because it's broken and people keep pushing they might get used like pushing reviews that might get used to that the bots like the build server fails so they will ignore it and this comes to the last point laziness people have the tendency to either rely completely on this so even review passes and stuff like this they sometimes don't necessarily review very carefully the review because they rely on the build server to catch up box and those kind of things so this needs to be clear set of policies to fix those issues so this is what we experienced but all over all this has solved so many problems and this has made our life at being easier so yeah let's move on to the last slide because I'm actually very curious to know what you have to say about this or how you experience your QA pipeline if you want to have a chat with me you just twit tweet me or DM me on Twitter and we can have a conversation about this because I'm looking forward to hear more about this or learn more about this yeah that's pretty much thank you very much for your time