We had to build a historical version of the original ad data analyzer that allows customers to build on the same custom pivot table reports using historical data.
Quickly processing that amount of data was the biggest challenge. With the ad data analyzer MVP, there were between one and two million rows of data being analyzed. With the historical ad data analyzer that amount jumped to 34 million. It is basically an archive of Catalist's data going back more than a decade.
Understanding the development challenges of dealing with this amount of data, I needed to design a product that would help diminish that data concern and make the product usable for clients.
User research was vital to understanding client needs. I conducted user interviews and user testing with actual clients to isolate the areas we needed to change or improve from the ad data analyzer for the historical ad data analyzer.
I heard that clients might occasionally look at one year's data in isolation, but often they needed to compare data between two election cycles. I also learned that users were interested in adding an additional filter beyond the four that were available. Clients talked about adding that additional filter at the end of the process, giving them one additional step at the end.
To help solve the big data challenge, I worked with our developers to design a few solutions that would not only improve the user experience but also cut down on the amount of data that needed to be processed. That helped to improve the load speed, but when load speed may still bel long, warn warn users so that the experience met their expectations.
When I dug deeper into how clients were comparing historical data, I learned that they needed to compare data from a full election cycle rather than specific dates. That allowed us to aggregate data for an entire year rather than needing to keep individual records for each day, greatly limiting the amount of data that needs to be processed.
