Today I want to talk a little bit about some of the underlying technologies we use in netDeFlect, Esphion's network anomaly detection solution. There are some core technologies, such as neural networks, which enable us to detect network anomalies by means of a highly-trained, specialized artificial intelligence, which is present in each netDeFlect installation. Other core technologies include components for high-speed packet processing, sophisticated data structures for fast lookups of information and advanced visualization and reporting capabilities.
To provide a complete, powerful and flexible solution, a lot of different technologies had to be combined in an innovative way. One of the things I find very attractive, but which normally takes place entirely hidden 'behind the curtains' is our use of load-balancing for the CPU intensive task of analyzing network packets and the extraction of zero-day signatures.
Before joining Esphion, I worked in a company, which specialized in server load-balancing solutions. I was an early employee there, and developed much of their core technology. Some of the largest web-sites in the world used our load-balancing systems to keep their business running, and manage massive amounts of hits on their servers. Therefore, I am excited that at Esphion we were able to put load-balancing again to good use.
Extraction of fine-grained signatures
After netDeFlect's neural networks detect an anomaly, the system then aims to provide signatures of this anomaly to the network operators. This has to happen fully automatic, and within seconds of the onset of the anomaly. Having those signatures then allows the creation of filter instructions for various network devices. I talked here about how our solution, in combination with already existing network infrastructure, can result in a network security architecture, which is surprisingly resilient even against zero-day attacks.
It is important to note that we are talking about true zero-day signatures, which in no way rely on prior knowledge or a signature database. Using several sophisticated algorithms, we can rapidly extract those signatures out of life packet samples, thereby enabling network operators to instantly get a handle on such anomalies, and to instantly proceed with the mitigation of the anomaly.
The signatures we provide need to be fine-grained to the extent that it must be possible to filter out the bad traffic with only minimal or no impact on the legitimate traffic (see here for more information on this requirement). Since anomalous traffic may at first look exactly like ordinary traffic, this is not a simple task: The algorithms to perform this analysis are quite CPU intensive.
Balancing the signature extraction work-load
When our solution is installed in a customer's network, there is
usually a centralized controller, as well as a number of agents
(sensors, in effect), which are distributed across the network.
After an anomaly has been detected, and after all the necessary data and packet samples have been correlated, it would be easiest to simply run the signature-extraction algorithms on the central controller. However, such a design would not be scalable.
Esphion always has had a focus on innovation, which should be reflected in the practicality and usefulness, but also elegance of our solutions. Therefore, in order to accommodate the often CPU intensive extraction of anomaly signatures, our engineering team has implemented an underlying load-balancing mechanism. So, when there are traffic samples that need to be analyzed for signatures, the controller will properly distribute the work among the installed agents in the network. Thus, the overall work-load is split among the agents, who all collaborate under the supervision of the controller to arrive at the required fine-grained signatures.
As a result, our solution has a built-in scalability, since it grows with the network in which it is installed. I think we have a powerful, elegant and intelligent architecture, which benefits the network operators, because they always get zero-day signatures within just seconds of the onset of an anomaly, no matter how big their network is. This scalability allows us to analyze anomalies faster and in much greater detail, thereby discovering information about anomalies that otherwise would remain hidden.
Juergen
Comments