Development, verification and testing of Digital Marketing outcomes with Machine Learning on Google Cloud Platform.

The media agency of one of Australia’s larger financial institutions required a Discovery Analytics environment to support an internal Marketing data initiative. Servian consultants designed a GCP Cloud based system which provided an appropriate and secure user experience for Data Analysts, Architects and Admins.

The Challenges

An Analytics environment

A cloud data analytics environment was needed for exploration of digital interactions as well as the optimisation of discovery analytics and conversion attribution modeling. This environment would need to include the ability to provide a greater level of insight into digital clicks and other interactions. The initial environment has poor query performance so the challenge is to increase performance and accessibility while providing additional functionality with Google Cloud Platform (GCP) and Google Cloud Datalab.

Secure Access for different roles

The Architect, Analyst, Admin and other roles all required daily secure access to data as well as SQL and Python programmatic access for transforms and other functions. The challenge is to ensure appropriate access controls for partners and staff.

Daily ingestion from multiple sources

The data is ingested from Adobe Analytics including FloodLight Custom Variables and DoubleClick Campaign Manager. It must be stored in a persistent manner and ingested daily. The data needs to be accessible from outside the institution environment. The ingestion process currently has a high development cost and the challenge is to provide ingestion of both data flows while minimizing the cost.

Our Solution

The Platform Design:

The Cloud Data Analytics Environment was designed using the Google Cloud Platform (GCP). By combining GCP cloud storage with their local data warehouse BigQuery this system allows for easy access and processing. To constrain the permissions the GCP Cloud IAM tool was used to restrict what each identity could access and use. The Platform was designed using the following tools:

  • Google Cloud Storage & Storage Transfer Service for Adobe data
  • BigQuery & BigQuery Transfer service for DoubleClick data
  • TensorFlow
  • Compute Engine
  • Cloud IAM
  • DataLab

Performance Testing:

Servian performance testing showed that the GCP user accounts could provide secure access for all identities and data was made available from all sources. By using links to Datalab the Machine Learning functionality was achieved and overall costs were lowered for data ingestion and storage.

Next Steps:

With no significant cost for the development ingestion process in the GCP environment, the Cloud Data Analytics Environment can now be scaled to use cases beyond the initial scenario.  Expansion of machine learning and analysis techniques will also be possible and future development can take advantage of the large suite of tools available on GCP (such as their Cloud Machine Learning Engine, BigQuery Data Transfer etc).

2018-09-19T23:57:37+00:00