This solution assists migration of a large private cloud to GCP platform. Project includes many solutions for monitoring, logging, migration of kubernetes clusters from OpenStack to GKE and much more.

  • Migration of Metric information from customer’s stack driver environments into a central monitoring system. This includes AppEngine pulling monitoring information from thousands of gcp orgs, processing them in DataFlow, publishing them in PubSub and storing them in BigQuery datastore
  • Migration of Logs from all clients into a central system
  • Architecture and design org security structure in relation to Identity Manager
  • Provide CI/CD solution including Google apps (e.g. Cloud Deploy, Deployment Manager) and non-google apps (Terraform, Spinnaker)
  • Providing Network solution in regard to client platform

Initial Phases

Migrate COLD & ADHOC clusters’ HDFS data to GCS

Automated deployment of Hadoop Clusters on GCE,

Management framework for users and groups, that sits on top of GCS ACLs

Redesign job and workflow tooling to support object store and new GCE cluster(s)

Support hourly on-premise hadoop data ingest to GCS

Develop change management framework to prepare and support teams using the GCE cluster(s) backed by object store


Technology Stack:

  • Hadoop Infrastructure on GCP
  • Tensorflow
  • Dataflow
  • BigQuery
  • Dataproc
  • Presto