While intrusion detection systems are becoming ubiquitous defensesin today's networks, currently we have no comprehensive and scientifically rigorous methodology to test the effectiveness of these systems.This paper explores thetypesofperformancemeasurementsthataredesiredandthathavebeenusedin the past. We review many past evaluations that have been designed to assess thesemetrics.We also discuss the hurdles that have blocked successful measurementsin this area and present suggestions for research directed toward improving ourmeasurementcapabilities.