Information about conference
TestCon was a three day conference held in three different locations in beautiful city of Vilnius in Lithuania. It started with a one full day workshop and continued with two session days. Workshops were held at the first day of the conference and you need to sign-up beforehand to reserve a seat. If you are late as I was, you might end up missing the most interesting workshops since they seem to be fully booked quite quickly. Workshops were organised in two different locations and you have to check the schedule from www.testcon.lt website to see where you need to go. One big thing to complain is that conference material was published quite late at their web-page after ten days. However, all sessions were recorded and uploaded to YouTube.
Session days were held in a location called Multikino, a big movie theater, at a mall in Vilnius. Workshop locations and Multikino were really far from each other so you cannot book a hotel which is in walking distance to all three. You will have to use a means of transportation to reach the venues.
Each session day was really busy having seven sessions, each 45 to 50 minutes. There was no lunch breaks. There was a time slot in the programme called “Lunch Break”, but since there were sessions during that time, you had to skip those sessions to be able to eat. Speakers were from all around the world and most of their main expertise was Quality Assurance. All of them were successful examples in their fields and they shared their experience about the tools and methods they have been using. It was hard towards the end of the day to digest the all the information flowing in, so at that point, I had to write extensive notes and pictures and went back to those after the conference.
I will go through with the main topics I was able to attend in this order: Test Driven Development, Test Automation, Quality Process.
Test Driven Development, TDD
Test Driven Development is a development method which almost every developer know by heart but not applied if it is not strictly required by the organisation they work in. TDD was main topic of few sessions and all agreed that applying TDD makes a big impact on the quality of the produced software as well as the culture of software development in an organisation.
Tests, in this context unit tests, are a kind of feedback about the code, similar to reviews. In spite of code reviews, tests are reliable and instant. When TDD is used, code emerges from the tests. Changes done to the code are always verified by the test and behavior of the code is always tested against the expected behavior. These are all known by us that behavior is testable with unit tests. Francesco Strazzullo who was talking about TDD in the first talk of the conference made a good point about TDD that, it is actually possible that produced code could also be tested against non-functional requirements (-ilities) by using TDD.
Unit testing is at the core of the TDD. Dennis Doomen talked about how to practice TDD and unit testing without shooting yourself in the foot. His approach was so that developers should not try to apply DRY principles to unit tests and it is okay to copy-paste a test to create another test with minor modifications. His definition of “unit” in unit testing seemed like a bigger concept than individual classes. This definition targeted testing a bulk of classes which enables a feature for the software, something we call system testing, in this case performed completely by the developer.
Finding and fixing bugs early in development is the cheapest way to deal with them. Test driven development helps finding bugs early in the development process. This was one of the learning of the conference: “Fail fast, fail early, fail often”.
Test Automation
Let’s first start with the definition. What is test automation? Test automation is a testing concept where set of tests run to use the produced software like an end user does. Automation performs certain steps and behaviour of the tested software is then verified with different tools such as comparing screenshots and evaluating the logs if applicable (in some sessions “Gray Box Testing” was mentioned so that test environment has some information about the tested software state to detect and deal with flakiness). Test automation is a crucial part of the modern software development process. Having automated tests dramatically reduces testing time and increases the reliability of the software. As a result, modifications to the software could be verified against the expected behaviour and as a result “continuous release” could be possible. Test automation is important to achieve scalable software development when cross functional teams produce their code and merge to a common branch, i.e. staging or even master. As there are so many positive sides of test automation, it is not so easy to achieve automated tests which are fast, reliable and easy to maintain. Automated tests are also a good way of verifying requirements of the product.
Jani Haapala, talked about how to perform scalable test automation. I have to mention that this was my favourite session in the whole conference and it was really eye-opening. He first emphasised the importance of picking right infrastructure. This heavily depends on what kind of software is produced as well as the capabilities of the involved people. Complexity of the problem grows when considering the possible layers for infrastructure and the tooling, so having the correct infrastructure is really hard and should be discussed in a separate post. Considering how hard this is, infrastructure and tooling could also be provided from an external party, and development of tests could be done in the software teams. This approach clearly defines the responsibilities and the contract between the parties. Jani also presented how they approach the failing tests. Flakiness is also a problem in their world and they created a mechanism to deal with the failing tests with different reasons.
Automated tests could be developed with different methods. I will write about three methods highlighted in the conference. First approach is developing a test framework for the product and maintain it. People can develop their tests by using the test framework and test framework could be maintained by a separate team. This approach requires everyone to be coders since developing tests is same as writing code. Second approach is using frameworks such as Robot on top of your test framework. Jani Haapala promoted Robot as it does not require much coding skills and readability of the tests is high. Readability is important when requirements are directly executed as tests. No need to say that almost no coding skills are required to develop automated tests with Robot Framework. Note that this requires considerable coding effort to first build a test framework and then adapt it to Robot Framework. However, it pays off when developing the tests with less effort than implementing the test. Third approach is using record/playback method to create tests. This approach requires a tool which the user (tester in this case) can start recording a test and performed actions are recorded as code. Generated code could be re-run as an automated test. It is fast to create automated tests with this approach. A person with no coding skills can generate hundreds of tests quickly. However, this approach will generate a lot of duplicate code and once product changes, it is likely that there could be huge amount of tests to be recreated. Louise Gibbs presented Record/Playback approach and she emphasised that refactoring generated tests was required to re-use some of the tests. She explained that a non-coder tester might not have adequate coding skills to create automated tests but he/she can learn while performing the refactoring on generated code. This approach could be beneficial if an organisation want to boost their automated test coverage but do not have enough coding power or time to do it.
Quality Process
Quality is by definition tells how good or bad something is. Sounds simple. Quality process is the set of rules, methods executed by actors whose goal is to have the product matching the specified requirements. In a big software project, it is not easy to manually verify if product is matching the requirements especially if product is created incrementally. An independent structure or team is needed to perform relevant actions to make sure that the quality targets are achieved. It is also important to note that, the mentioned verifications below should be done by that independent party who do not have a conflict of interest with the development teams.
To achieve quality, test automation is crucial in the modern software development world. If the produced software has test automation directly testing the functional requirements, incremental modifications are not as scary and minimal amount of manual work needed for testing. This will require functional requirements to be defined in smaller sets, or “user stories” and each story should have at least one automated test which will tell if a previously implemented requirement is still met after the changes to the product. Independent quality organisation should verify the tests against the defined requirements.
Having automated tests directly running the functional requirements will enable the ability to have a release anytime in a product timeline with periodical release targets. This approach makes things scalable for cross functional teams and also makes things much easy when some parts of the product is outsourced.
Test driven development is also an integral part of the quality process to find problems and deal with them in early phases. It could be utilised to check non-functional requirements.
One important aspect is the actors for managing the quality process. Conference had the message that “Everyone is responsible for quality” and it is a culture for software development organisations. I recommend watching the session from Torstein Skarra From Waterfall to Agile: 7 tips for Transforming Test Organization.
I will have a separate blog post about software development and how to achieve the quality which will have my further learnings from the conference.
Thanks for reading.