The age-old adage “if it ain’t broke, don’t fix it” often serves as a deterrent to embracing automation. Many organisations are hesitant to disrupt existing technical workflows, especially if they’ve proven effective, if not efficient, over time. However, this mindset can also be a barrier to progress and innovation, and in some cases, costing time and therefore money.
While certain processes may appear functional on the surface, they may still harbour inefficiencies or limitations that could be addressed through automation. By challenging the status quo and being open to change, organisations can uncover hidden opportunities for improvement and unlock new levels of efficiency and effectiveness.
This blog post will explore how even seemingly “unbroken” processes can benefit from automation, demonstrating the value of taking a proactive approach to innovation, as highlighted by a recent automation project with a manufacturing company in an industry that dates back to the 19th Century.
“By focusing on objective evaluations of workflows and processes, stakeholders can identify areas for improvement and implement solutions effectively”.
An open-minded, collaborative approach is essential for successful automation projects, especially in network management. By focusing on objective evaluations of workflows and processes, stakeholders can identify areas for improvement and implement solutions effectively. This approach was exemplified in a recent manufacturing company’s successful automation project caried out by KedronUK, where a clear focus on process evaluation led to significant improvements.
Initially, ten workflows were identified for evaluation.
The goal was to:
• Identifying the stakeholders involved in each workflow.
• Thoroughly understand the current workflows.
• Quantify the time and effort involved in each workflow.
• Assess the feasibility of automation of each workflow.
Each of the ten workflows were assessed and categorised for feasibility, efficiency benefit and cost verses benefit. It was found that the initial ten workflows fell into the following three main categories:
1. Mitigating Ticket Proliferation in IT Service Management:
There was a need to address the issue of an excessive number of tickets being generated in the IT Service Management Platform. Streamlining and refining the ticketing process would be pivotal in enhancing overall operational effectiveness.
The Network Operation Centre (NOC) team, where finding it very difficult to stay on top of the 10,000 plus tickets being generated through existing integrations.
2. Worlkflow Automation for manual Tasks:
An automation opportunity was identified within manual workflows to eliminate redundancy associated with repetitive tasks. This included expediting the onboarding of new devices and the cessation process for existing devices. By automating these procedures, the aim was to enhance efficiency, reduce errors, and accelerate the overall pace of operations.
3. Efficient Ticket Generation for the NG Firewall Platform:
Automating the process of ticket generation for the Next Generation Firewall platform, with threat intelligence, to ensure a swift and accurate response to detected threats and assessments. This would involve integrating automation solutions that expedite the identification, logging, and resolution of issues on the platform, ultimately contributing to a more responsive and agile operational environment.
Lets look at these in turn:
Mitigating Ticket Proliferation in IT Service Management
Three tiers, or components were involved in raising ITSM tickets for the organisation, which within themselves already had a degree of automation implemented. The results however had become unmanageable, with approximately 10,000 tickets per month being raised for the 24×7 NOC team to triage and close. This equated to roughly 14 tickets per hour around the clock.
Upon investigation, it was discovered that a significant portion of these tickets were duplicates or repetitions of similar events, leading to a staggering 70% increase in ticket volume. The existing automation had become inadequate, exacerbating the issue rather than resolving it.
The first tier, the Network Management tool, had root cause calculation capabilities but was configured to forward all alarms, without root cause, to the second tier—a Network Management tool with integrations to the third tier, the ITSM platform. While this setup seemed promising in theory, it proved ineffective in practice, as evidenced by the overwhelming volume of tickets inundating the NOC team.
The immediate and pressing question arose: Why wasn’t a tool equipped with root cause analysis capabilities being fully leveraged? The answer, though somewhat surprising, revealed that the second-tier solution possessed the capability to filter—not correlate—for alarms tagged with a root cause from the first tier. Furthermore, the business had decided to only address root cause incidents of specific types via the proactive team in the ITSM, with the remainder managed by the Business as Usual (BAU) Team through reports. Consequently, this criterion was also added to the filter.
This setup meant that, regardless of the configurations in tier one, tier two would only forward what it was configured to, resulting in the decision to send everything from tier one to tier two.
We recommended reversing this logic, making the more capable tier one tool the one with the intelligence to determine what to send after calculating root cause. This approach would leave tier two with the straightforward task of merely forwarding what it receives. Additionally, this approach simplifies future configuration changes, as there is only one tool to configure.
This change resulted in a 33% reduction in the number of alarms sent to tier two, all of which matched the proactive team’s criteria. However, the number of ITSM tickets remained roughly the same.
In delving into the root cause of the ticket surge, we examined a month’s worth of ticket data. Our analysis revealed a trend: a substantial number of tickets were being closed by the proactive team, marked as acceptable within business utilisation thresholds. Moreover, we observed a proliferation of seemingly duplicate incidents, where multiple tickets were processed and closed by the team, referring to existing open tickets.
The investigation yielded two significant recommendations. Firstly, we proposed fine-tuning the tier one management platform to trigger alarms based on business utilization thresholds, which notably curtailed the number of utilization related ITSM tickets.
Secondly, we investigated the issue of apparent ticket duplication for identical incidents. We uncovered a limitation within the tier two platform—its ticket-raising process lacked an update mechanism. When a condition resolved to its KPI, an “OK” notification was issued from tier one to tier two. Tier two would then locally close the incident without updating the ITSM. This oversight led to recurrent breaches generating new tickets. The oversight was blamed on the business requirement for all tickets to be closed manually.
A solution was needed to update open tickets with both the “OK” notification and recurrent breaches. However, we hit dead ends with the tier two solutions capabilities and with the ITSM platform team due to a reluctance to alter logic, therefore, we redirected our focus to tier one. Leveraging its capability to directly interface with the ITSM tier bypassing tier two, we achieved the required ticket creation and update process.
Overall, monthly tickets saw a remarkable 77% reduction, plummeting from 10,000 to 2,300. This significant improvement allows the team to allocate more resources to incident resolution rather than ticket deduplication. Furthermore, the business is now evaluating the business case for the tier two solution, with potential cost reductions on the horizon.
As can be seen, the insights gained from our analysis led to recommendations aimed at optimising processes and enhancing productivity. In the upcoming segments of this blog series, we’ll continue our dive into additional facets of network management automation with a look at Workflow Automation for Manual Tasks followed by Efficient Ticket Generation for the NG Firewall Platform.
If you would like to discuss an Automation or Consolidation project, please contact phil.swainson@kedronuk.com.
Chris Booth
Solutions Architect
Listens to your problems then identifies the best tools and products to build solutions.
Call us today on 01782 752 369
KedronUK, Kern House, Stone Business Park, Stone, Staffordshire ST15 0TL