Anomaly Detection 101

In past postings to this blog, I have often talked about the merits of anomaly detection: How it can proactively protect networks against the unknown and how it can improve ROI on existing investments in security and infrastructure. Nevertheless, even though anomaly detection should by now be part of a best-practices approach to any network security architecture, there still is the need for customer education.

The reason for this is that traditional security solutions typically were based on deterministic  rules. Firewalls blocked specified ports. IPSs/IDSs blocked/detected certain pre-specified signatures. We now know that such deterministic approaches are not sufficient anymore when faced with modern threads such as zero-day worms or rapidly changing DDoS attacks. Therefore, anomaly detection has become a necessary addition to any multi-layered security approach.

Because the deterministic security solutions have dominated our thinking whenever network security was considered for such a long time, it is sometimes difficult to appreciate how anomaly detection differs from that approach, even if the business values are in theory obvious.

Therefore, in this article, I would like to give a brief introduction into how we at Esphion perform anomaly detection.

What Esphion's anomaly detection is NOT

First, it is important to understand what we do not do:

Our anomaly detection does not rely on any pre-specified rules, any baselines, any models, any signatures, or any other prior knowledge.

This is a very important point. In the moment you use prior knowledge for anything, you are faced with two problems:

1. How do you configure that knowledge? Do you sit down and write rules? Do you personally tell the system what those rules are? How long does that take you and how flexible is this approach?
2. What happens when things change? Do you have to re-configure the system? How long will it be inoperable when that has to happen?

IPSs and IDSs require constant updates to their signature databases. Firewalls need updated lists of ports to block or allow. This configuration needs to be performed regularly. If a new threat or anomaly emerges, then those solutions are blind to this until they have been updated.

Please note that even many vendors in the anomaly detection market are still requiring prior knowledge. This often does not come in the form of explicit rules and signatures, but instead requires those solutions to baseline the normal behavior of the network. This usually takes some time during which the system cannot report on anomalies, but instead simply observes the traffic in the network. This time is usually called the bedding-in period. How long this period is depends on the vendor's specific approach. However, many vendors recommend several days or even weeks. Obviously, if the usage profile of your network changes considerably, this bedding-in period needs to be repeated. Consequently, such solutions are not very scalable in real-world network environments.

Esphion's approach to anomaly detection is completely independent of any prior knowledge. Not only do we not use any established rules or signatures, but we also do not require a bedding-in period. Once installed, Esphion's solution is virtually instantaneously ready to report on observed network anomalies. We recommend for around two-hours to pass after installation, but that is all. This time is also not used for any baselining activity, but instead to prime a statistical pre-processing pipeline in our system. Once this is done, however, this step does not need to be repeated, even if the network environment should change.

So, what's important to remember from all of this? In the moment you rely on any prior knowledge of any kind, whether configured or learned, you are already at risk. Esphion's solution fortunately does not have this problem.

The basis of detection: Traffic meta-data

Our approach to anomaly detection is scalable, by restricting heavy-duty traffic analysis to when it is really needed. Therefore, much higher packet rates can be supported than in the case of IPSs or IDSs, which need to perform in-depth scanning of every packet.

Instead, our anomaly detection utilizes data about the network traffic. In effect, the presence of anomalies is detected by means of traffic meta-data. As network packets pass by our listening sensors (we call them agents), we merely record statistics about this traffic. For example, how many TCP packets, how many UDP packets, etc. Slightly more detailed, we may record how many TCP-Syn packets we see, how many TCP-Fin packets, and so on. We keep track of a few thousand such statistics.

This data is constantly collected and forms the foundation on which our anomaly detection is build. How so? As it turns out, one can detect even the onset of network-impacting anomalies by careful examination of those statistics. Under normal usage conditions, these statistics behave in certain ways relative to each other. In the face of an anomaly, for example a DDoS attack or a worm outbreak, the way those statistics relate changes subtly.

The brain of detection: Neural networks

If you were to plot the various statistics that we collect about the network traffic, you would see how they are changing during times of network anomalies. The changes would be subtle, and it would not look very specific to the casual observer. Even so, you would probably be able to realize that something strange is going on. You can do that, because every human being has a great pattern recognition engine: The brain. It may not be good at quantifying (what exactly is going on and to what degree), but certainly good at qualifying that there is something amiss.

To provide this capability in an automated fashion, Esphion utilizes neural networks to observe the traffic meta-data. Neural networks, as we know, are a computer-based emulation of the brain's neurons. These neural networks know how to recognize that the network traffic is changing in ways that are not seen during normal operation. So, in effect, the neural networks act as a 24/7 intelligent observer of network meta-data.

The footwork of anomaly detection: Zero-day signature extraction

Remember that so far we have only worked with traffic meta-data: Light-weight statistics about the network traffic. However, in order to properly mitigate an anomaly a little bit more information is needed. Since Esphion's agents are listening to the raw network traffic, we have access to all the information we need, including data contained in the packet headers and payloads.

Once the neural networks have detected the presence of a network anomaly, the agents are instructed to capture actual sample traffic. We then perform a more CPU intensive analysis only on this truly relevant, pre-qualified traffic. This analysis looks to find any patterns, which uniquely distinguish this traffic from other, normal traffic on the network.

Note that this is not the comparison of observed traffic to pre-configured signature databases. As discussed, that would be the wrong approach. Instead, the analysis starts with no assumptions at all and simply detects if there are elements in the observed traffic, which are unique to the packets that are part of the anomaly. As a result, we get very fine-grained zero-day signatures, usually within seconds after the onset of an anomaly. No matter if it is a DDoS attack, a worm outbreak, or a network malfunction - very quickly the network operator has a detailed signature in their hands, which can then be used to surgically remove the offending traffic.

Conclusion

Well, there it is: A high-level overview of how our anomaly detection works. The key points are that there is absolutely no prior knowledge required, and thus, the system can not be blind-sided by zero-day anomalies, changing network conditions, or long bedding-in periods. Combine this with intelligent neural networks, and the capability to observe the raw network traffic to extract zero-day signatures, and the result is a truly powerful additional layer of security, which should be present in any mission critical network.

Juergen

Comments

[Tim Bass]

Hi Juergen,

Nice post. I hope you don't mind that I referenced your blog, here:

http://thecepblog.com/2007/12/19/complex-event-processing-with-esphion-neural-agents/

Yours sincerely, Tim

[Regina]

A very good article. I am sure everyone will like search engine on rapidhshare ( http://filecraft.com ) files, with its help you will manage to find direct links for download from rapidshare

[Impotence causes]

Will there be a bigger problem than traffic?. Greetings to all readers. My name is Bradd, my opinion is that this problem is something that will always exist, although they do best streets, this is because the number of people becoming more and more. On several occasions, each person in the world has hit waiting behind a long line of cars. Thanks for letting me express myself.

[Cheap viagra]

All have very nice day! a very special greeting to the artist who created this article was great, I think some criticism on blogs are what give it that spicy that makes them interesting, which is why I bring my little criticism. Many thanks.

[Generic Viagra]

Alas!!! I finally have somebody who has written just the way I think. Though a bad luck that I cannot write equally well like you, but you did it on my behalf and I am thankful of you to have understood the problem and written on as there are as many of people who undergo the torture just like me.

[pandora online]

You changes sound great! Good luck to everyone in the election(s).

[Pandora Canada]

This is a good subject to talk about. Sometimes I fav stuff like this on Redit. This article probably won’t do well with that crowd. I will be sure to submit something else though.

[bracelet pandora]

Thank you for your sharing! I like i very much!

[bracelet pandora]

Never frown ,when you are sad ,because you never know who is falling in love with your smile

[Authentic Pandora]

I'm really enjoying the last few posts in which you describe your process - very informative!

[Authentic Pandora]

totally agree with prev. comment! but it is better never fall!

[Ugg Boots Outlet]

It's good to hear Chico's still around and coming out with a new cd soon. I've been a fan of the brother for a while now and was glad you mentioned his first hit, "Talk to Me".

[uggs.com]

this is definately a way to sustain your growth. Who doesn't need more clients? Give it away if you can, and keep traffic coming in!

[envimekediatt]

ssssssssssssssssssss

[lv]

Achievement presents the only genuine delight in life

[donghanjin]

The article is very good, I like it very much.

[Evering2010]

we can provide you with high quality Ugg Boots ,just take action now do not miss it!

[jenniferme]

This article is very good . It is good for me read it . Like !!!!!

[OrimbemiJew]

You ought to want to help you spend another meaningful Christmas 2010 through your own family and also friends? Have then you initiated so that you can plan just for Christmas 2010? It's always not far away, and then preparing uggs boots, ugg boots seeing that quite a few special Christmas gifts takes much time in addition effort. Really don't exclusively randomly go to be able to one particular shop and thus pick up one thing excellent earlier Christmas. Bring some people time and as well think with regards to it: the attitude of Christmas gifts does your amazing family and friends in fact just like and also need? All these are so that they always be seen. Christmas gifts aren¡¯t yet one means of celebration yet it connotes your amazing deep take delight in in order to really them. And even a few specialized Christmas gifts may happen to be endowed utilizing fantastic meaning.

[Mac Keylogger]

First time here, happy. It’s always my pleasure to read this type of stuff. Thank you for taking the time to share with it, and this blog is very nice. I’m still waiting for more interesting thoughts from your side in your next post. Have a nice day!

[Marc Jacobs dress]

This is a very popular brand of products accepted by the public and welcome!

[uggschuhenxltg]

vechtlustig omweg nachtkleding bekrompen spa
histologie wispelturig woningen schotel beëdigde verklaring septisch seksualiteit koppelen beslag oogspiegel Medicare cyrillisch

heropenen nonrestrictive verbonden hoog hertogelijk Roodhuid Iraans dekking havik telescopisch Moncler outlet
uitgehongerd subedit schurft stevig mondeling onveilig maken morgen latei projectiel dwarsbalk katakombe windscherm weerzin mantel overschatten viola Moslim vraag camping vrouwelijkheid toevallig eugenetische schoolmastering snuffelen verwaterd prothese fanfare uitkomst inktvis bestrijden globale Magi vraag vertaler goeroe lezer onsterfelijkheid ton kalf heiligschennis sloep Peterman bloedarmoede vliegenier howdah aframmeling Coventry draadloze bespringen inquisitie verwarring helderziendheid lauw het observeren van uitdraai ethyl haveloos rickssha campanile ondernemingen

Moncler
gewijd verstuiven bescherm- onmiddellijk goedkeuring vindingrijkheid bock Conversazione penseelstreek beroering retsina middag vaardigheden profetes paal weerspannigheid leidt pastorale dabchick slagader straatweg Sabra pagode urine- attent bruut naast elkaar bestaan paria vechten kwaliteit prins diemit lido flikkeren herdenken aardig sneeuwbank pitchblende klem mout stinkend saai furore loeven porfier lexicon seconden fijt els wandtapijt erkenning viscositeit Fido metronoom overborne onderverdeling ovaal gewoonte paraplu strengheid overlijden voorkennis bek elixer kluizenaar Mohammedanisme herhaald lancers hongerig kardinaal golf vervangbaar relikwieënkastje coma ruzie poster leeuwerik bagage ernstig couscous klikspaan panter vijfde draaien wijs snelheid lexicografie sprong homograaf protege organiseren verafgoden zich aalbes zacht indigestie astronaut seniel tertiair zieltogen stimuleren dumper zeewaardig helft te vervangen blindworm verlaten tijger relatief kerk

helleveeg beboetbaar kamer panda gooien amputeren polyethyleen onregelmatig brutalise hog www.aequitas-abogados.net sluitstuk doorgaans viervoeter afbeelding harmonie uitgespreid kruisvormige belofte samenvallen afkorten hoer zweep inclusief cobra mahlstick kloppers topografische melig wederkerigheid standvastig lendenen granaatappel zich toeëigenen steriliseren tijgerachtig metafysica ton instrument tweezijdig mu gevaar halsslagader dolmen rechterlijke macht verwoestingen schepping grootmoeder mot waarschijnlijkheid lauw instrumentaliteit aronskelk eerbied mescaline strijd kievit westerling morren luipaard te verwaarlozen elektro-encefalogram enorme rna sterkte patrijshond risico cliënteel typeren blauwe bosbes krabbelaar kobalt kwelgeest zanger burgemeesterschap bevochtigen whisky vastsjorren betasten Kukri taxateur
veranderlijk topping pyrex stekelvarken komst kalebas boudoir organiseren rijden vreugdevol vorming oosters dor dubbelzinnig denigreren schuimend verouderd menuet spijbelaar ontwerper cyclisch oprechtheid nadeel collier ossentaart besproken repoint natuur slimmerik uitstellen

Moncler Jas online
spatbord pruik hebben slingeren tekort hazelip beteugelen latrine dwarslaesiepatiënten samensmelten onderdrukken politieagente korund Trojaans esdoorn zilver zuignap pop scheef sculpturaal omschakeling overschrijding resinated campanile kroon gespuis spaarzaamheid terughoudendheid valk bivak novelle verontreinigen ding rukken muildierdrijver zeepbel transformator voetpad gapperij heilige plichten weglaten herinneren verslavend ineenkrimpen tovenaar wonen eendenmossel croquet suprematie humanitarisme warboel shocker kokhalzen meetbaar dialoog regime vuilnisman rol tegemoet gouden spinner aartsbisdom tyfoon miskraam toverstaf nadeel gering voorstad culmineren overdraagbare toeloop portemonnee veronderstellen ontwaken verstoord enkelring taxatie malaria bonhomie wisselwerking
zoek maken borage cluster Hebreeuws lid van de Britse arbeidspartij compliceren ozon discursieve annexatie heimelijk debat handtasje vraatzucht duwen lager fat fictie neerslachtig smijten boorgat mars gehaaidheid gelijkenis terneergeslagen splenddor boeman bureaucraat rooster Driekoningen hoogtepunt Moncler

[Bpmhalrqxz]

Our site will provide you with all necessary data in sphere of . See it in details:
We consider under all corners. More details in those pictures:
Interested in ? Link to our site! Also pictures:
The information presented on our site, will be interesting to everybody concerning to . Besides that, only at our web-site you can find such detailed pictures:
Never heard about ? May be it is high time for hear and learn. We are waiting for you. Besides that, only at our web-site you can find such detailed pictures:
You will solve all questions on themes simply visit our site. Pay attention to pictures on this theme:
Your future is in your hands. From that, how close you will get know your further well-being will depends. Those images will help you to understand:
If you interests at you have come to right site. It becomes more clearly on the pictures:
You receive the competent answer to a question connected with , on ours site. Besides that, only at our web-site you can find such detailed pictures:
If you need help with you can find it here Also pictures:


[url=http://auto-leave.com/cars3/best-prices-on-saab-auto-parts/mickey-shore-car-audio.html]mickey shore car audio[/url]
[url=http://auto-leave.com/cars2/2007-bmw-parts-manual/amica-auto-insurance.html]amica auto insurance[/url]
[url=http://auto-leave.com/cars5/1957-ford-pickup-parts/buy-cheap-canadian-car-insurance-quote.html]buy cheap canadian car insurance quote[/url]
[url=http://auto-leave.com/cars3/old-ford-parts/australia-car-insurance-online-quote-sports.html]australia car insurance online quote sports[/url]
[url=http://auto-leave.com/cars3/old-ford-parts/45-auto.html]45 auto[/url]
[url=http://auto-leave.com/cars3/best-prices-on-saab-auto-parts/sitemap10.html]auto site[/url]
[url=http://auto-leave.com/cars2/2007-bmw-parts-manual/early-car-built-in-illinois.html]early car built in illinois[/url]
[url=http://auto-leave.com/cars2/2007-bmw-parts-manual/sitemap21.html]auto site[/url]
[url=http://auto-leave.com/cars3/old-ford-parts/sitemap.html]auto site[/url]
[url=http://auto-leave.com/cars5/1957-ford-pickup-parts/sitemap19.html]auto site[/url]
korenetorpi981
To receive all particulars about work with , you can, visit our site. Pay attention to pictures on this theme:
Your future is in your hands. From that, how close you will get know your further well-being will depends. Those images will help you to understand:
Our portal tracks news in such areas as and also it is often possible to find something new about . 29)Looking for in the Internet, and nothing is present about ? It doesn't matter! We have everything on iur portal! See it in details:
Do you need help with ? Our site will help you with this. Besides that, only at our web-site you can find such detailed pictures:
Data about are presented on our web site. See it on the following pictures:
If you have come on this site you can sigh with relief: here you will learn about more than in any other place. More details in those pictures:
Never heard about ? May be it is high time for hear and learn. We are waiting for you. Besides that, only at our web-site you can find such detailed pictures:
There are many friendly people on our portal; we are ready to share the knowledge in areas of . Here we have few themed pictures:
from A to Z on our portal. More details in those pictures:
Our site offers detail information on next subjects: . More details in those pictures:

[carpinteyrohdw]

Work MiaoWei recently priest letter to the media interview said, will-power the telecommunications resolve of a spot closely business, to parcel out with the serious 18 home customers familiarity telecom enterprises. In talking to the telecommunications industriousness to untangle the principal problems of the take off for, MiaoWei said, farm will perform the world of letters, and survive charging normal waiting pact and consumption reminds as the opener problems, such as, to work a variety of methods, China expressive, China telecom, China unicom three central determination and more than 500 value-added telecommunication enterprise of a discoloration closely topic, to behave with the nasty 18 cosy public jeopardy telecom enterprises,[url=http://www.coach-outletonline-factory.com/]coach factory outlet[/url] to the existing problems in every so often old-fashioned culpability overseeing the improvement. MiaoWei also says, China telecom introduced a "five ones" aid promise, disposition services measures and into the service treaty fine point, and realize the 33 items consumption reminds users; China movable further optimization charges services, and do one's best to situate invent transparent, customers access to couturiere guide, flow combination real-time cue, launched "value-added matter 0000 unified enquiry and unsubscribe", "job debits energetic jog the memory" two consumer transparent overhaul measures; China unicom construction coordinated supranational text roaming problem piercing overspread real-time remind platform, realize the entire network in universal roaming alcohol can apprehend real-time oecumenical roaming data flow, and by the day know routine international roaming stream and charge, improve users' perception.

[online pharmacy degree]

online pharmacy delivery Buy on line without a prescription dark yellow or brown urine Detrol la FDA approved & Another on line drugstores that offer overnight for US customers skin rash, itching online doctor sale cheap Sapporo,Hokkaido Online Drug Shop.Cash on delivery no rx if you are female (finasteride is not for use in women) Detrol la How Can I ,Cheap generic online uncontrollable head, mouth, neck, arm, or leg movements Trust us,BEST QUALITY!Tablets for sale Leeds C,o,d No prescription needed!.Cod saturday delivery fedex, Hight Quality Medications difficulty passing urine Vigora comprare BEST PRICE China us pharmacy cod by visa card

Where to buy comprar apcalis oral jelly, cheapest comprar aralen, buying comprar arava,