Safeguards with Google Cloud Functions

More Tutorials
Safeguards with Google Cloud Functions

Safeguards allow you to define rules for when Experiments can or cannot run. This tutorial shows how you can use a Google Cloud Function as a safeguard.

Safeguards with Python Cloud Functions

ChaosIQ has a feature called Safeguards. When Chaos Toolkit is connected to ChaosIQ it will check the safeguards before executing any step in an experiment - this gives the ChaosIQ user the ability to stop experiments running at any stage. Currently, Safeguards are implemented as webhooks. ChaosIQ calls the webhook and if it responds with {"status":"ok"} experiments will continue. If the webhook responds with {"status":"ko"} experiments will stop executing.

Google Cloud Functions

Google Cloud Functions is a serverless execution environment for building and connecting cloud services, see this quickstart for all the steps required to write and deploy one in python https://cloud.google.com/functions/docs/quickstart-python. In this article, we will explore using a Google Cloud Function written in Python as a Safeguard in ChaosIQ.

We are going to write a Google Cloud Function that can be used as a Safeguard for the Chaos toolkit when it’s connected to ChaosIQ.

The source code for the Google Cloud Function can be seen under the Chaos toolkit Community Playground Cloud Function

The Cloud Function is triggered from an HTTP request as follows: https://us-central1-gcloud-py.cloudfunctions.net/hour_in_range?lower=9&upper=12 If you call the URL at 11, it will respond with {status: ok} whereas if you call it at 13, the function will respond with "status":"ko"}

Chaos Console

If you have signed up and are running the ChaosIQ console you will be able to view your experiments on the ChaosIQ dashboard, if you have yet to sign up you can sign up for a free trial here ChaosIQ. A screenshot of the ChaosIQ console showing completed executions can be seen below:

Chaos Console Dashboard

In the above screenshot, you can see my latest experiment run with a completed status, I will now create a safeguard that will block completion by using the url:

https://us-central1-cloud-sib.cloudfunctions.net/hourInRange?lower=9&upper=10 (the current time is past 10 am)

Chaos Console Safeguards

Now I attempt to execute my experiment again, see the screenshot from my terminal window. Note the console window shows the safeguard has disallowed the experiment from continuing.

Chaos toolkit console

I can also see the ChaosIQ console also shows the interrupted state of the experiment

ChaosIQ executions interrupted

Safeguards are a mechanism you can use to give you full control over your and your teams experiment execution environment. If you are running in a shared environment whether its staging or production having experiments running at random times or running at the same time as each other could have a big impact on your results, that is where safeguards come in.

If you have yet to use the open source Chaos toolkit you should give it a try. If you want the extra control of safeguards then you should try the ChaosIQ SaaS then please sign up for a free trail and give it a go ChaosIQ Free Trial.