Getting Started with Automation

Getting started with correct picture of Automation process matters a lot, whether you are looking for CLI or GUI automation.
Although I will be taking about TCL/Python automation, this automation process is applicable universally for all languages.

I have led automation teams of Cisco and TCS. I have delivered complex automation tests on routers, switches and servers which includes CLI and GUI automation. Hence, let me share with you a little insight on automation process for a successful automation.

Lets start with a pictorial view of end to end automation process.

We can form 3 teams which can work independently and deliver successful automation.

Framework Developers : Main task of this team is to provide base libraries which can connect to your DUT (Device Under Test), Logging, Error handling, protocols for writing testcases, reporting, etc. This team is the crux of automation and usually same framework can be used across all the teams in the company. Hence, this can be a totally independent or a centralized team.

Library Developers / Testcase Developers : Library development and Testcase development is a close-nit team, hence its better that one team takes this activity. We always develop procedures/functions/API that suit our testcase needs. This team develops testcases on top of the framework produced by Framework Developers. Testcase developers also need good insight on the technology they are working on and are generally in tight sync with QA (Quality Analyst) team or Manual testing team.

Operations Team: Last but not the least is the operations team which should take care of tests scheduling, analyzing and reporting needs of production level automated tests. This team has number of tasks as given below :

• Preparing test suites (a batch of testcases to be run)
• Maintaining testbeds (hardware or environment where the tests will run)
• Web Page reporting, creating database driven web pages.
• Preparing execution servers
• Developing scripts to launch automation through CLI / GUI. 
• They can also do innovative work such as automated rerun of failed cases, developing analysis tools, etc. 

The above teams are good to have for complex automation, however we can merge operations with testcase development team if the tasks in hand are small.

We will dig more into details of each team and automation details in the subsequent posts . Posts are still being written, please be patient.

Do let me know if you have any particular queries.

Overview of End to End Automation

Lets see how End to End Automation can be divided into functional layers.

For end to end automation, first we have to determine our DUT (Device Under Test) and the interfaces through which we can access our DUT. For plain software products DUT is nothing but software itself.

Generally, we use Browsers , Desktop Applications UI (User Interface) , CLI or Mobile Apps as the interface. Hence, as per our needs we need to have libraries to connect , disconnect, send cli commands, parse output, start GUI tests, retrieve results, ......etc. These libraries are customized according to our DUT or testcases. Hence, these libraries are written by library development / testcase development teams.

For GUI, we might be using Ranorex / Selenium / TestNG or similar frameworks with their own coding languages like C#, VB.Net, Java , Python, however, we need to create wrapper functions for end to end automation to take place. These wrappers will be written by library development team.

Automation Framework

When I say Automation Framework, I don't intend Ranorex / TestNJ or similar frameworks. I am talking about TCL/Python automation framework. For GUI applications , I will wrap their framework inside TCL/Python automation framework to get power of TCL/Python and their language. It also gives me one launching script for both CLI / GUI operations.

Throughout my blog my assumption will be using TCL/Python over Linux/Unix/Sun kind of OS. I don't think TCL/Perl/Python can be used in their full potential on Windows.

Earlier, I was involved in automation of Cisco's routers/switches which had only CLI interface. Although there are management software which gives GUI interface for managing routers/switches, but world over lab admins rely mainly on CLI.
Few years later, I came to Cisco UCS Servers group where I needed to have a powerful framework which could support both CLI and GUI execution. GUI automation is never simply GUI focused, we still need CLI / XML access to verify GUI status. Hence, fusion of scripting languages with GUI automation frameworks makes a lot of sense. 

Automation Framework will give us generalized procedures or packages which we can use to develop our customized libraries. It also takes care of robust error-handling and reporting.
Example, we just need to call a proc to connect via ssh/telnet and it should take care of exceptions and generating various log files adding time-stamps to logs.
Medium / Big organizations have already adopted some automation framework which provides these functionalities.

• Main components of Automation Framework are :
• Self sufficient installation package
• Well defined directory structure for
• Libraries
• Test Files
• Job Files / Test Suites
• Config Files
• Topology files
• Output / Report collection
• Pre-written Packages
• Binaries / Scripts to launch automation (with a huge list of optional arguments :) )

If you want to know more about this, then I will share some samples later while describing automation framework in detail.

Execution using Automation Framework

Every Automation framework comes with binaries / scripts which will help launch the automation.

For execution of tests we need following things:

• Job File or Testsuite File
• Config File
• Topology file (optional)

Job File / Test Suite File : This is a file which tells which all tests to be run as part of this job / suite. We can create a complex job file depending upon our requirements or we can have a simple job file which just specifies the TCL/Python scripts to run with their parameters.

Config File : Here, I should introduce a concept of testbed. A testbed in terms of automation is all the devices which will participate in automation. E.g. In case of CLI automation a testbed may consist of 2 routers (DUT), 1 traffic generator, 2 Linux servers.
A Config file(s) will capture all the details related to these devices which might be necessary while running a test.

Once a testbed configs / topology files are written, we can replicate that testbed configs with changes in IP addresses, names, etc. This will enable us to run same job on different testbeds.

We can either write classes for testbeds and in config files just instantiate the objects using different parameters or we can have plain config files. Its up to you if you want OOP (Object Oriented Programing) way of writing files or not.

It makes sense to have a common config file where we generally store parameters which are common across all testbeds. Eg. can be storage locations, NTP server IP, DHCP server IP, etc

Topology File : The interconnection between these devices can be captured in some topology file (XML is good choice of language). This is necessary when we have large set of interconnections between devices and our tests needs the details of these interconnections. With such topology file, we can also generate dynamic pictorial views of our testbed. For some automation tests this file might not even be necessary.

Scheduler

Test schedulers help us scheduling our job / testsuites on our testbeds. I also help us prioritize certain jobs dynamically so that we are not stuck in FIFO (First In First Out) execution.

With help of schedulers we can choose which jobs should run on which testbeds using some execution servers. It can have its own logs storage apart from automation framework. 

Schedulers can also provide some mechanism to pause an ongoing job, although it might use automation framework intelligence to pause the jobs. Hence, they are not completely isolated from automation framework and they are generally built with some automation framework in mind.

Schedulers generally have some web access, were we can see testbed queues , running job status and other functions described above.

Generally, in schedulers we have one to many relationship between job/testsuite file and testbeds. This enables schedulers to queue jobs on multiple testbeds and the testbed which becomes free can start running next job in queue.

Reporting

Reporting is the last thing which happens when the job gets finished. We can send reports using emails or we might update some database and create a web dashboard where users can view reports as per their convenience.

Reporting on webpages will need some knowledge of dynamic web languages like PHP. Once, enough data is pushed in a database related to a job, we can manipulate this data and present interesting facts and figures which are sometimes missed. 

Example usage of reporting are 

• We can see average run time of all our jobs
• We can see testbed usage or average idle time
• We can see how many automation runs are happening over a period of time
• We can see which tests are more stable than the others
• We can check which tests are catching more bugs

One important part of web page reporting is to have some kind of analysis page also available for the executed jobs.