Event-Driven Scheduled Jobs Using GCP Cloud Tasks
In our company Interviewer.AI, we have been using a lot of AWS Lambda / GCP Cloud Function for scheduled tasks. For example, a daily event scheduled via AWS Cloudwatch Event to Lambda in order to send out daily insights / reports for our users.
So far the setup works for us pretty well.
Recently, we started to work on a product feature where we wanted to send out email reminders for our candidate at a specified future time (think of it as X hours after the candidate applied for a job). While the original workflow with scheduled cron timing still works to a certain extent, there will be unnecessary scanning of DB and processing for information that might not be needed.
Since each email to be sent is tied to an “event” when a candidate entry is being created, this is a good use case to use an event-driven + asynchronous architecture. We started thinking about using some sort of queueing service, however, we quickly realized there is a downside of normal queueing or publisher/subscriber service such as AWS SQS or SNS as the services usually don’t have a “temporal” element of it. Unless you publish the message to the queue say 48 hours down the road, the message will be consumed straight away by the downstream services.
This is when we were introduced to GCP Cloud Tasks by a Google Customer Engineer. In a nutshell, it’s a task queue service that comes with the feature we need the most in this problem statement, i.e. scheduling. Aside from that, other benefits that come along include task de-duplication, configurable downstream rate-limiting, and can store messages in the queue for up to 30 days.
Summary of the implementation as below:
