+Advanced Search

Conformal Anomaly Detection Method for Unstable Logs
Author:
Affiliation:

Fund Project:

  • Article
  • |
  • Figures
  • |
  • Metrics
  • |
  • Reference
  • |
  • Related
  • |
  • Cited by
  • |
  • Materials
    Abstract:

    System logs are used as the primary data source for system anomaly detection.??Existing log anomaly detection methods mainly use log event data extracted from historical logs to build detection models, that is, the distribution of log data is assumed to be stable over time.??However, in practice, log data often contains events or sequences that have not occurred before.??The instability comes from two sources: 1) conceptual drift occurs in logs;??2) noise is introduced during log processing.??In order to alleviate the problem of instability in logs, an anomaly detection model called Ensemble-Based Conformal Anomaly Detection (EBCAD) based on confidence degree and multiple algorithms is designed.??Firstly, the p-value statistics are used to measure the non-conformity between logs, and multiple appropriate ensemble algorithms are selected as the non-conformity measure functions to calculate the non-conformal scores for collaborative detection.??Then, an update mechanism based on confidence is designed to alleviate the problem of log instability. By adding scores of new logs into existing sets, the experiences of log anomaly detection are updated. Finally, according to the confidence degree and the preset significance level obtained by collaborative detection, the unstable log is judged to be abnormal.??The experimental results show that when the unstable data injection rate increases from 5% to 20% in HDFS log data set, the F1-score of EBCAD model only decreases from 0.996 to 0.985.??In the BGL_100K log data set, when the unstable data injection rate increases from 5% to 20%, the F1-score of EBCAD decreases only from 0.71 to 0.613.??This proves that EBCAD can effectively detect anomalies in unstable logs.

    Reference
    Related
    Cited by
Article Metrics
  • PDF:
  • HTML:
  • Abstract:
  • Cited by:
Get Citation
History
  • Received:
  • Revised:
  • Adopted:
  • Online: May 13,2022
  • Published: