How To Handle Long Running Tasks In Ansible

Ansible is an automation and configuration management tool that can be simply used to automate different tasks and for provisioning and configuring complex cloud infrastructures. You can kick start your organization's Ansible journey in no time. 

For this blog's purpose, we assume that you have a basic understanding of Ansible and how it works. We will specifically discuss the tasks which take more time to complete. By default, for Ansible tasks, the connections stay open until the task is done on each node, but this may not be the desired behavior for some functions as sometimes the task can take more time than SSH timeouts. 

You can run such long running tasks in the background and check their status later. You can use the async_status module to track the status of the job running by passing it the job ID that was returned when you ran the original job in the background.

Pre-requisites

It would help if you had Ansible installed (preferably the latest version) on your system, from where you will run your playbook. We will explain below a hypothetical task for database import; if you want to test this method, you can use any such task for testing purposes. It would also help if you had to make sure that you have configured the Ansible managed hosts according to your testing requirements.

Ansible Playbook

Below we have an example of Ansible playbook to import MySQL database dump, and it also performs some other tasks on the managed node (dbhost in this example). 

Now, there are mainly two tasks you can focus on in this example playbook: Start database import and Check if database import is complete. 

We assume that the database import task takes enough time to complete. By default, Ansible will wait for this task to complete and will only proceed with the rest of the tasks once this one is complete. In some cases, you can even face SSH timeouts if it takes more than the specified timeout.

So, rather than waiting, what if we can jump to some other tasks and finish them in the meantime and let this run in the background? That way, we can get rid of timeout issues, and reduce the overall time the playbook takes to complete.

One thing to take into account is - you should only run those tasks during this time that are irrelevant to the database import or which aren’t performing any action that needs the database to be fully available. Once you are done with all other tasks, you can get back to the import task and check if it's still in progress.

Playbook Execution

So let’s explain each module/parameters we used in these tasks to understand how it works:

TASK: Start database import

Name: Name of the task
Shell: your command to run
Async: async is used to explicitly set the timeout you wish to apply to this task. Rather than relying on the connection method timeout, we can have a custom timeout for a particular task. The async tasks will run until they either complete, fail, or timeout according to their async value.
Poll: poll is a value for which the playbook will stick on the task until it either completes, fails, or times out. For long running asynchronous tasks, it’s good to set poll=0 so that Ansible can immediately jump to the next task after starting the current one without waiting for the result.
Register: It is basically used to register the results on a task in a variable. If we want to evaluate those results later or retrieve something from the task results, we can use the registered variable. Here it is vital because this variable will be used later to check if the task has been completed or not.

TASK: Check if database import is complete

Name: Name of the task
Async_status: async_status is used to check the asynchronous task status later, which you ran earlier. You can use the async_status module by passing it the job ID that was returned when you ran the original job/task in the background. This module takes a parameter jid where you can pass the job ID of the task you want to check the status
.ansible_job_id variable stores the job ID for the job/task since we registered a variable for our task earlier so we can retrieve/use the job ID and can pass to jid like this jid: "" where import_db is the variable we retired for our database import task.
Register: here, we are registering another variable job_result to store the results of the current job/task so that we can check the status of our database import task is completed.
Until: it will keep on checking the task until it is finished or gets a timeout.
Poll: according to the interval we have set for the poll, it will keep on the polling status of the task. For example, if poll=10, it will try to check the status of the task after every 10 seconds, if it met the required state of what we have defined in until.
Retries: the number of times you want to retry to check the status of the job until it gets timeout or declared as failed. For example, if you have set retries=20, it will try to check job status after every 10 seconds (if poll=10) 20 times. In short, it will wait for the task to complete for 200 seconds in total.

Below is the execution of the playbook:

The Start database import task started the database import, and rather than waiting for the database import to complete, Ansible jumped to the next task. It finished TASK1, TASK2, and TASK3 in the meantime. Later at Check if database import is complete task, Ansible started polling the database import task to check if the import is done.

This is one of the examples to handle a long running task in Ansible. There can be tasks that take a couple of minutes to complete. So, rather than waiting for that task to complete, you can let that run in the background and proceed with the next tasks. Later, you can retrieve and check results for that particular task to finish without any issues.

Hope this tutorial helped you in managing your Ansible tasks in a better manner! 

Did you know Ansible and Docker go hand-in-hand? Find out how you can manage your Docker using Ansible!