AdHash
25x time and 43x cost reduction using Buglab.The Customer
AdHash offers a new approach to digital advertising by rethinking the RTB processes and providing in-depth analytics while keeping the netizens' privacy, well private. It optimises the entire workflow for both Publishers and Advertisers, removing unnecessary middlemen, minimising overhead, and reducing the campaign launch time from weeks to minutes. Quite a complex solution to fit into a few words.
The Challenge
Building two complex platforms with hundreds of pages in two different colour themes is hard on its own. Developing and enhancing them in rapid succession while ensuring the best UI and UX for customers became a huge challenge as the platforms expanded internationally.
The scope in numbers
2 client platforms with more than 100 pages each:
- both in light and dark themes;
- both on desktop and mobile screen sizes;
- Breaking down to 260 tests for the Advertiser platform;
- And 319 for the Publisher platform.
Totalling around 450 pages to be tested only for one device with one theme. Multiplied by 8 for all major test setups and we have a bare minimum of 3,600 tests. Add a new release every few weeks to the mix and the numbers are adding up pretty quickly.
A conservative estimate would look like this:
- 3,600 test;
- A new release approximately every 4 weeks;
- The average time to execute 1 test is 5 minutes.
3,600 * 5 = 18,000 minutes or 300 hours every 4 weeks. This is the required time to test only the visual side of the AdHash platforms. Functional testing is not included in the calculation.
Testing with Buglab
To address these challenges and reduce time and overhead, AdHash ran a test with Buglab. Starting from scratch the initial set-up took 10-12 hours including:
- Set-up;
- Project and Suite creation;
- Individual test creation with 10-60 commands each;
- Test runs, reviews, and corrections;
The result
8-10 hours saved by the "Clone" function for setting up similar tests for the light theme and mobile version.
A full-scale weekly test costs less than $100 per month compared to the monetary value of 300 hours of manual human work.
The low barrier to entry and low/no-code interface enables non-technical staff to set up and perform the testing. No need for rare and expensive talent to create and maintain the testing environment.
Quotes from the AdHash team
"Some minor UI changes may affect up to 80% of the pages in the platforms. To identify and test every single page where those changes take effect I would need at least 32 hours (about 3-4 hours per platform). With Buglab, it takes me about 3 hours to run all 220 tests and review them. And I have the full confidence that no pixel on any page was left unchecked."
"Non-UI sensitive issues were discovered. During data migration, discrepancies in some reports were picked up. We were able to recover the lost data from the tests screenshots, and correct the reports."
"We use Buglab not only for UI testing but UX and functional testing too. Including but not limited to: ensuring redirect links work as expected; the correct data is rendered when opening specific pages; data for different date ranges are calculated and plotted correctly; creative previews are rendered correctly; etc."
In conclusion
"By using Buglab as our primary visual testing solution we've reduced the time it takes to ensure we deliver the best user experience by 25x and the testing costs by 43x. Buglab helps us in lots of corner cases and hard-to-spot data discrepancies. The platform's value outmatched our initial expectations and continuously helps our dev team focus on building new features without ever worrying about breaking old ones. In addition, the "click" command enables us to include a bit of functional testing to our visual ones without writing a single line of code. The raw HTML comparison feature helped us with MVC optimisations and bug fixes. We highly recommend Buglab for development and QA teams at all levels."
< May 2023 - Buglab platform version 0.1 (alpha) />