Patrolling network traffic with SecBI

Network Traffic Analysis tools have been used for a long time to help improve efficiencies in enterprise networks, locating unused capacity and bandwidth, and eliminating chokepoints. It has recently been employed as an arm of cybersecurity too. That makes sense given that, except for insider threats, attacks are going to be initiated and ultimately controlled by outside elements. The communications between the internal threat malware and its controllers on the outside are captured by traffic analysis tools.

The problem is that while the logic of using traffic analysis in cybersecurity is solid, the reality is a bit different. For one, even a small to medium-sized enterprise is going to generate three or four billion traffic logs per month. Without computerized assistance, no human is going to be able to wade through that and find anything meaningful. Second, capturing all that data traditionally requires the installation of network traps on gateways across the network. For an organization with branch offices or remote locations, the number of trap installations can climb pretty high. And even then, some traffic may escape around those gateways.

SecBI has fielded new software that aims to eliminate both of those problems, volume processing of data for actionable threat intelligence, and a reliance on network tapping hardware. They have done this by deploying their analyzer as a software module capable of running on-premises or in the cloud. It only looks at log files, so there is no need for any network taps, agents on the clients or anything beyond access to the constantly generated log files. It then crunches those billions of events in the log using finely tuned algorithms that look for patterns associated with an ongoing attack or an advanced persistent threat (APT). It can be deployed with a pay-as-you-go contract, where users only pay based on how many gigabytes of log file data they need to process per day.

To test SecBI, we began working with a version of the program that had been installed locally and was collecting data from the system log files of a test network for several months. Various malware and APTs had been seeded within the test network to give us something to examine.

Looking at the main console interface, we could see that SecBI had already identified an active campaign within our test network that was comprised of 2,555 events. You would think that with so many things happening, that it would be easy for an analyst to see that as a red flag. However, looking at individual items showed why this was evidence of a hidden threat moving very slowly and avoiding standard detections.

John Breeden / IDG

Each of the pings leaving the test network from the malware went to a different URL. They were all very long, obviously machine generated URLs. However, the referer hosts were masked. If only looking at a single event, an analyst might conclude that YouTube or Yahoo or Google had sent a user to an address with a long string, which can sometimes happen. The host names were similarly random and only used once by the threat actors. They would not have tripped any threat intelligence feeds based on prior use by malware.

The outgoing traffic was also extremely spread out over a two-month period, at seemingly random times. It did very little in terms of file transfers, sending out only about two megs of data overall and pulling in only seven megs over that same months-long period. An extremely alert or very lucky analyst might somehow connect a few of the dots and think that something was up, but this low and slow approach is why APTs are able to remain undetected inside networks, especially very large ones, for months or even a year on average before getting caught. Many APTs are only caught when they make a huge data exfiltration move, their likely goal the entire time, and by then it’s too late to do anything about it.

SecBI, however, is designed to piece together those seemingly harmless, random events into a story that shows what is really happening. It does this by running all those billions of log events through algorithms looking for patterns. The SecBi intelligence examines many factors including the repeating structures found within URLs, the times of the reach out calls, the frequency of the events, whether or not the URLs are machine generated, the real or masked referers, and the type of traffic going in or coming out of a network. 

John Breeden / IDG

In the case of the test APT, those seemingly random strings of letters and numbers in the URLs were not truly so random. SecBI was able to determine a pattern that was used within the randomness. The threat author also used the same rotating masked referer hosts. The threat did a few other quirky things, like always sending out two pings, three hours apart, and then resting for a long period of time before going again.

Armed with that level of data, the SecBI program successfully went through the entire system log files and located over 2,000 other events that fit any of those patterns. It then assembled its collected data in a graphical chart showing exactly what was happening over time, plus all of the clients that were involved.

John Breeden / IDG

We were amazed at how easy it was to see evidence of an attack when presented this way by the SecBI software. It’s doubtful that any single, or even a few, individual elements would have looked overly suspicious on its own. We are talking about a few URLs mixed in with billions of lines in the log file. The best comparison might be the difference between being presented individual words in seemingly random order and being given an actual finished article to read.

Once we had all the events collected into a chronological story, as administrators we had a couple of options. First, we could label everything in the report as benign. Perhaps our company is behind the strange URLs going out as part of some unique program. If that is the case, setting it as ‘not a threat’ will cause SecBI to stop concentrating on it. Those processes will still be monitored, but no more alerts will be generated unless the pattern radically changes, which might be an indication of a new compromise.

More likely, admins will want to mark everything in the report as malicious. Doing that confirms to the SecBI software that it did indeed find evidence of an ongoing attack. Most administrators will then start disconnecting and scrubbing infected hosts to eradicate the threat. SecBI will thereafter keep monitoring the situation and alert if a supposedly clean network suddenly begins sending out data following the same, now-known malicious patterns. This acts as a confirmation that either everything has been fixed or that more work needs to be done.

John Breeden / IDG

Traffic monitoring could be a great tool to use against APTs and stealthy malware, since the most advanced ones will need to contact their hosts from time to time for more instructions. It’s just extremely easy for attackers to hide those communications within the billions of other events happening within most networks. SecBI can unmask those hidden threats, while rounding up the whole picture for cybersecurity teams, forming useful order out of the huge, chaotic system log files that everyone is already collecting.