Automating Data Integration Across the Digital Marketing Ecosystem

Overview

Client: Online marketplace

How we helped an auction listing company to automate the collection and consolidation of data from multiple marketing and analytics platforms into a centralised data source. The solution reduced manual reporting, minimised data-entry errors, and gave the business secure, on-demand access to consistent, up-to-date insights.

Business challenges

Staff time lost to manual reporting: Valuable staff hours were spent collecting, checking, and consolidating data instead of focusing on higher-value work.
No single source of truth: Data was scattered across multiple platforms and spreadsheets, making it difficult to maintain a reliable, up-to-date view of performance.
Limited visibility for decision-making: No consolidated view of key digital metrics, such as advertising spend and revenue, website engagement, SEO, app growth, social media, and email performance. This made it harder to quickly assess performance and make informed decisions.

Technical challenges

Inconsistent source data: Each platform (Google Play, Google Ads, Brevo, LinkedIn, and more) exposes data differently, with its own schemas, metrics, formats, and API behaviours. Each source therefore required a tailored ETL pipeline to clean and standardise the data.
Managing different update cycles: Platforms updated their data at different frequencies, with some metrics changing retrospectively. The ingestion process needed carefully designed schedules to ensure reporting remained consistent and current.
Reliable data orchestration: Multiple pipelines had to run automatically, handle dependencies and failures, and reliably transform and load data into the existing MySQL database without disrupting the reporting layer.

The solution

We replaced a manual, spreadsheet-based reporting process with an automated data integration and analytics platform. Data from multiple marketing, advertising, website, app, social, and communications platforms is automatically fetched, processed, standardised, and consolidated into a central MySQL database.

The consolidated data is presented through a secure Flask dashboard, giving the client different views of their digital performance without requiring staff to manually compile reports. Users can access pre-built reports covering areas such as advertising costs and revenue, clicks and CTR, website and SEO performance, app downloads and registrations, social media growth, and email performance. Reports can be filtered, sorted, viewed on demand, and exported to Excel.

The dashboard provides a consistent view of the underlying data, while the automated ingestion process ensures that scheduled updates are processed without staff intervention. This shifted the reporting workflow from manually collecting numbers to accessing and analysing a centralised, maintained dataset.

What we built

API integrations: Built integrations with Google Ads, Google Analytics, Brevo, App Store Connect, Google Play, LinkedIn, and the client’s own application database to reliably extract data from multiple sources with different APIs, schemas, and reporting structures.
Python ETL pipelines: Developed source-specific pipelines to clean, transform, standardise, validate, and load extracted data into a consistent structure suitable for reporting and analysis.
Scheduled ingestion: Containerised the pipelines and deployed them to AWS ECS, using cron-based schedules tailored to the refresh cycles and historical update behaviour of each data source.
MySQL data layer: Loaded processed data into MySQL on the client’s existing AWS RDS infrastructure, providing a centralised and queryable data store.
Flask analytics dashboard: Developed a secure Flask application with secure authentication, pre-built reporting views, filtering, sorting, and Excel export functionality.

Results

100% automated data collection: API data is fetched, processed, and loaded automatically on a predefined schedule, removing manual data entry from the reporting workflow.
1 centralised reporting view: Replaced fragmented reporting with a single dashboard covering advertising, SEO, website, app, social, and email performance.
Only ~R400/month hosting costs: Right-sized infrastructure avoided the higher cost and operational overhead of a larger business intelligence platform.
Staff time redirected: Eliminated recurring manual reporting, freeing staff to focus on higher-value work.
Lower error risk: Removed manual transcription and spreadsheet consolidation, significantly reducing opportunities for data-entry and reporting errors.
On-demand reporting: Internal users can access and export the latest reports as needed, without waiting for reports to be manually prepared.

Fathom Data’s philosophy

We believe the best technology solution is not always the biggest or most feature-rich one. Our goal is to understand the client’s actual requirements, including data volume, number of users, reporting needs, and existing infrastructure, and choose the most appropriate technology capable of solving the problem well.

We considered several ways to build the dashboard, including React, Vue, and business intelligence platforms such as Apache Superset. With fewer than five internal users, these options would have introduced infrastructure and operational complexity that wasn’t justified by the requirements. Flask provided everything the client needed in a lightweight, maintainable application that was quick to develop.

This approach also delivered a measurable cost benefit. The Flask application can run comfortably on a small AWS instance such as a t3.micro or t4g.micro, and the full platform costs around R400 a month to host, far less than the larger infrastructure typically required for a platform such as Superset.

For us, this is an important part of good engineering: build for the problem, not for the technology.