The Platform Design:
The Cloud Data Analytics Environment was designed using the Google Cloud Platform (GCP). By combining GCP cloud storage with their local data warehouse BigQuery this system allows for easy access and processing. To constrain the permissions the GCP Cloud IAM tool was used to restrict what each identity could access and use. The Platform was designed using the following tools:
- Google Cloud Storage & Storage Transfer Service for Adobe data
- BigQuery & BigQuery Transfer service for DoubleClick data
- Compute Engine
- Cloud IAM
Servian performance testing showed that the GCP user accounts could provide secure access for all identities and data was made available from all sources. By using links to Datalab the Machine Learning functionality was achieved and overall costs were lowered for data ingestion and storage.
With no significant cost for the development ingestion process in the GCP environment, the Cloud Data Analytics Environment can now be scaled to use cases beyond the initial scenario. Expansion of machine learning and analysis techniques will also be possible and future development can take advantage of the large suite of tools available on GCP (such as their Cloud Machine Learning Engine, BigQuery Data Transfer etc).