Marketing

Goals

• Deploy application in to Pets at Home GCP Account
• Ensure application was working at all steps
• Troubleshoot any issues
• Ensure security best practices are followed

Technology used

Project Breakdown

I was approached by an external vendor asking for me to review some code that was written. After reviewing their code under the impression that this was a weekend project of theirs, I gave them some tips and approval that it looks good.

Later it was brought to my attention that this was a deployment the vendor was trying to make in to our Google account.

I was passed the task of refactoring this code to be repeatable across environments, as well as to generally re-architect it, so it works.

What I did?

I personally was responsible for:

• Upgrading the helm chart to:
  • Enable us to add IAP login
  • Ensure cron jobs were working (Concurrency with Sidecar containers is a headache)
  • Adding in Cloud Armour support
  • Adding in support for Vulnerability patch updates
• Security
  • Disclosing security vulnerabilities to the vendor
  • Ensuring the application was patched
  • Guiding the vendor on how to patch the issue
  • Testing the patched issue to ensure it was no longer present
• Writing the Terraform
  • init
    • Project creation
    • API Enablement
    • State bucket Init
    • Reserving static IPs (Due to internal IT Limitations)
  • Infrastructure module
    • VPC
    • Service accounts
    • Private GKE cluster
    • Private Cloud SQL Cluster (HA)
    • NAT Management
    • Cross project IAM
    • Workload Identity
  • Application Module
    • Authenticating to GKE with no local files
• Running the application
  • Ensuring the application is hitting the SLa standards that are set internally
  • Ensuring the underlying GKE cluster is up-to-date and secure
  • Ensuring the underlying other services (memcache,sql) are secure and up to date

Issues I had to overcome

• Vendor security issues
• Training Team member
• Cost management
• Learning Kubernetes
• Learning Terragrunt

Vendor Security issues

During the Internal infrastructure development, I was editing Account keys in the web interface, when I saw I was able to edit and view the keys in plain text.

This was already somewhat worrying to start with, but after accessing the private SQL Database (Creating a custom docker image) and checking the database I was able to locate the keys stored in plain text.

I created a write-up and explained how this could be abused, as well as what the fallout of this being breached could mean for the company.

Management put pressure on the external vendor to patch this issue.

Training Team member

As we are a small team, it means that we tend to have one person own a project to completion, which I know, screams 🌽silo

To do our best to break these down, we have catchup's weekly and sometimes daily (depending on the daily change volume) to allow other engineers to ask questions, as well as give design feedback before you're 2 weeks down a feature that is not really solving the original problem.

Secondly to the meetings, is ensuring that documentation is well laid out and in a simple-to-read manner.