Using a fully serverless cloud set-up, sometimes the easy questions like: "how can I run this function on a recurring basis?" are not so easy to answer. CRON job? nope.
Instead, this is how I achieve it utilising scheduled Cloud Tasks for a fully serverless approach.
Let's say we have a Cloud Run Function we want to call on a daily, or perhaps weekly, basis. The process we'll follow is:
- Set up a Cloud Tasks queue
- Set up the function to reschedule itself
- Create our initial task to set off the chain
Service Account
First, we create the service account used to run our cloud function
resource "google_service_account" "<my_sa>" {
account_id = "<my_sa>"
display_name = "My Service Account"
description = "Service account used for my function"
project = var.project_id
}
Then we add all required roles, specifically those for Cloud Tasks & invoking Cloud Run
resource "google_project_iam_member" "<my_sa>" {
for_each = toset([
"roles/run.invoker",
"roles/cloudtasks.enqueuer"
])
project = var.project_id
role = each.value
member = "serviceAccount:${google_service_account.<my_sa>.email}"
depends_on = [google_service_account.<my_sa>]
}
and finally, we also need to allow the service account to impersonate itself for queuing the next cloud task:
resource "google_service_account_iam_member" "<my_sa_self_impersonation>" {
service_account_id = google_service_account.<my_sa>.name
role = "roles/iam.serviceAccountUser"
member = "serviceAccount:${google_service_account.<my_sa>.email}"
}
Cloud Task Queue
Next, create a cloud task queue.
resource "google_cloud_tasks_queue" "<my_task_queue_nane>" {
name = "<my_task_queue_name>"
location = var.region
http_target {
http_method = "POST"
oidc_token {
service_account_email = <my_sa_email>
}
}
}
Queue Task from Function
Then, we can add a final bit of code to the end of our function to allow scheduling the next task. This is where we will set any rules like how long to schedule (1 day ahead, 1 week, etc), and what time of day to run at for example. Any shared data to be passed along must be passed as the HTTP body, and the function entry set up to parse this accordingly.
def cloud_task_scheduler(days_increment: int = 1):
try:
client = tasks_v2.CloudTasksClient()
parent = client.queue_path(GCP_PROJECT_ID, LOCATION, CLOUD_TASK_QUEUE_ID)
FUNCTION_URL="<my_cloud_function_url>"
# define next task
payload = {
"schedule_next_cloud_task": True
}
body_bytes = json.dumps(payload).encode("utf-8")
scheduled_date = (datetime.now(timezone.utc) + timedelta(days=days_increment)).replace(
hour=16, minute=0, second=0, microsecond=0
)
ts = timestamp_pb2.Timestamp()
ts.FromDatetime(scheduled_date)
task: tasks_v2.CreateTaskRequest = {
"http_request": {
"http_method": tasks_v2.HttpMethod.POST,
"url": f"{FUNCTION_URL}",
"headers": {"Content-Type": "application/json"},
"body": body_bytes
},
"schedule_time": ts
}
# add task to queue
client.create_task(parent=parent, task=task)
except Exception as e:
raise Exception(f"Error scheduling next Cloud Task: {e}")
Schedule the First Task
To set the chain off running, we must schedule the first cloud task. Simplest way is via the command line with Google Cloud CLI.
gcloud tasks create-http-task my-first-task \
--queue=my-queue \
--url="my-function-url" \
--method=POST \
--body-content='{"schedule_next_cloud_task": true}' \
--header="Content-Type":"application/json" \
--oidc-service-account-email="my-sa-email" \
--location="region" \
--schedule-time="YYYY-MM-DDTHH:MM:SS+00:00"
And just like you should see your first task sitting in the queue waiting to run! Of course, I reccomend testing first to ensure that your function can run fully and then create irs following task successfully.
This approach can be used for a daily schedule like I've outlined above or even to split up long-running tasks that may exceed timeouts by either passing shared data through the body of the task request or writing to a database
The benefit is because the cost of Cloud Tasks is basically nothing, or literally free if you are below certain usage tiers, then this method is extremely efficient as we only pay for compute while our function is running and it can effectively scale to zero while waiting for the next cloud task to run.
Hopefully this helps anyone else after a simple approach of implementing a self-sustainaing schedule with serverless compute efficiently!