Aposemat IoT-23

A labeled dataset with malicious and benign IoT network traffic

This dataset was created as part of the Avast AIC laboratory with the funding of Avast Software

Citation

If you are using this dataset for your research, please reference it as “Sebastian Garcia, Agustin Parmisano, & Maria Jose Erquiaga. (2020). IoT-23: A labeled dataset with malicious and benign IoT network traffic (Version 1.0.0) [Data set]. Zenodo. http://doi.org/10.5281/zenodo.4743746”

Download

Download the full IoT-23 dataset (21 GB) here:

• https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/iot_23_datasets_full.tar.gz

Download a lighter version containing only the labeled flows without the pcaps files (8.8 GB) here:

• https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/iot_23_datasets_small.tar.gz

Download the design of how the labels were assigned from this spreadsheet

• https://docs.google.com/spreadsheets/d/1HRqgKJp0XoSUIfW3rCQKoD_LnSCJ1k-k61PndJXWq_o/edit#gid=0

Introduction

IoT-23 is a new dataset of network traffic from Internet of Things (IoT) devices. It has 20 malware captures executed in IoT devices, and 3 captures for benign IoT devices traffic. It was first published in January 2020, with captures ranging from 2018 to 2019. This IoT network traffic was captured in the Stratosphere Laboratory, AIC group, FEL, CTU University, Czech Republic. Its goal is to offer a large dataset of real and labeled IoT malware infections and IoT benign traffic for researchers to develop machine learning algorithms. This dataset and its research is funded by Avast Software, Prague. 

The IoT-23 dataset consists of twenty three captures (called scenarios) of different IoT network traffic. These scenarios are divided into twenty network captures (pcap files) from infected IoT devices (which will have the name of the malware sample executed on each scenario) and three network captures of real IoT devices network traffic (that have the name of the devices where the traffic was captured). On each malicious scenario we executed a specific malware sample in a Raspberry Pi, that used several protocols and performed different actions. Table 1 shows the characteristics of the IoT botnet scenarios and Table 2 shows the protocols that were found in each network traffic capture. The network traffic captured for the benign scenarios was obtained by capturing the network traffic of three different IoT devices: a Philips HUE smart LED lamp, an Amazon Echo home intelligent personal assistant and a Somfy smart doorlock. It is important to mention that these three IoT devices are real hardware and not simulated (see Images 1,2 and 3) . This allows us to capture and analyse real network behaviour. Both malicious and benign scenarios run in a controlled network environment with unrestrained internet connection like any other real IoT device. Table 3 shows the network data of the IoT benign scenarios and Table 4 shows the protocols found in each network capture.

The goal of this dataset is to make the two types of datasets available for the community: the first type contains malicious network traffic and the second one benign IoT traffic only. Both benign and malicious traffic flows have two new columns for network behaviour description labels. These labels are assigned following the next process:

• The original .pcap file is analysed manually. The suspicious flows are detected and labels are assigned in an analysis dashboard.

• The labels were assigned by using the rules defined in this spreadsheet here, and our program Flaber. The labels were generated by an analyst.

• The Flaber python script reads the data of each flow in the conn.log file and compares this data with labeling rules. The script compares each flow with the rules and if the flow data fits the labeling criteria, the corresponding label is added.

Notice that the final labeled flows are in the files bro/conn.log.labeled for each capture.

Summary of the datasets

For each capture we provide a folder that contains the following files:

• README.md: this file has the capture and malware information such as the probable malware name, md5, sha1 and sha256 of the malware binary; the duration of the capture in seconds, the link to the VirusTotal malware file and some short description of the files inside the folder.

• .pcap: this the the original pcap file from the network traffic capture.

• conn.log.labeled: this is the Zeek conn.log file obtained by running the Zeek network analyser using the original pcap file. This conn.log.labeled file has the flows of the capture network connection as a normal Zeek conn.log file but it also has two new columns for the labels. Further in this document there is a list of the possible detailed labels with their description. 

• Other files generated that are explained in the further section Individual details for IoT-23 captures

IoT malicious flows dataset tables

In this section we will show a summary from the twenty malicious scenarios. Table 1 shows the scenario number (ID), the name of the dataset, the duration in hours, the number of packets, the number of Zeek IDs flows in the conn.log file (obtained by running Zeek network analysis framework on the original pcap file), the size of the original pcap file and the possible name of the malware sample used to infect the device.
Malware captures are executed for long periods of time. Due to the large size of the traffic generated by each infection, we rotate the pcaps files every 24 hours.  However, in some cases, the pcap file was growing too fast and we decided to stop the capture before the twenty-four hours were completed. For that reason, some of the captures differ in the amount of hours.

Table 1: Summary of the Malicious IoT Scenarios

To have some extra data regarding the network traffic generated by each infected device we used the application layer protocol prediction from Zeek to filter and summarize this information. In Table 2, this information is summarized, here, we included for each scenario, the name of the dataset, the amount of flows for the following protocols: HTTP, DNS, DHCP, Telnet, SSL and IRC. some protocols were not recognized by Zeek, there is a column where all this flows are quantified.

Table 2: Breakdown of Application Layer Protocols as detected by Zeek on the Malicious Scenarios.

IoT benign flows dataset tables

In this section we show tables with the network information for the bening scenarios. These scenarios had been created by capturing network traffic data of not infected real IoT devices. The column with the malware name was changed to specify the device name.

The bening scenarios are obtained by capturing the network traffic of real IoT devices. It's important to see and understand how real IoT devices behave in the network when they are not infected. This will allow us to identify a change in the behavior when they are infected with malware or are under attack.

Table 3 shows the network data for each one of the bening scenarios, including information regarding the duration, number of packets, number of Zeek flows, pcap file and the name of the device. Table 4 shows the application layer detected protocols for each one of the bening scenarios.

Table 3: Summary of the Benign scenarios.

Table 4: Breakdown of Application Layer Protocols as detected by Zeek on the Benign Scenarios.

Download the IoT-23 Dataset

There are two options to download the IoT-23 dataset. The first option is the full download, that includes the original .pcap, README.md and conn.log.labeled files which are a part of a bigger group of files for each individual scenario which are listed in Links to individual datasets in IoT-23. The size for the full version is 20GB. The second option is to download  a light version that only contains the README.md and the conn.log file. The size for this version is 8.7GB. Both options are available to download in the following links:

Full download link (20 GB): https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/iot_23_datasets_full.tar.gz

Small download link (8.7 GB): https://mcfp.felk.cvut.cz/publicDatasets/IoT-23-Dataset/iot_23_datasets_small.tar.gz

Also each capture can be downloaded separately as described further in this document in the section.

Explanations of the labels

To provide a more detailed information to network malware researchers and analysts, this dataset also contains labels to describe the relation between flows related to malicious or possible malicious activities. This labels were created in the Stratosphere laboratory considering the malware captures analysis. 

Here there is a brief explanation about the labels used for malicious flows detection based on the manual network analysis: 

Attack: this label indicates that there was some type of attack from the infected device to another host. Here we are labeling as attack to any flow that, by analysing its payload and behaviour, tries to take advantage of some vulnerable service. For example, a brute force to some telnet login, a command injection in the header of a GET request, etc.

Benign: this label indicates that no suspicious por malicious activities where found in the connections.

C&C: this label indicates that the infected device was connected to a CC server. This activity was detected in the analysis of the network malware capture because the connections to the suspicious server are periodic or our infected device is downloading some binaries from it or some IRC like or decoded orders are coming and going from it.

DDoS: this label indicates that a Distributed Denial of Service attack is being executed by the infected device. These traffic flows are detected as part of a DDoS ​​attack because of the amount of flows directed to the same IP address.

FileDownload: this label indicates that a file is being downloaded to our infected device. This is detected by filtering connections with response bytes more than 3KB or 5KB, normally this is combined with some known suspicious destination port or destination IP known to be a C&C server.

HeartBeat: this label indicates that packets sent on this connection are used to keep a track on the infected host by the C&C server. This was detected by filtering connections with response bytes lower than 1B and with periodic similar connections, normally this is combined with some known suspicious destination port or destination IP known to be a C&C server.

Mirai: this label indicates that the connections have characteristics of a Mirai botnet. This label is added when the flows has similar patterns as the most common known Mirai attacks. 

Okiru: this label indicates that the connections have characteristics of a Okiru botnet. This labeling decision was made with the same parameters as with Mirai but with the difference that this botnet family is less common.

PartOfAHorizontalPortScan: this label indicates that the connections are used to do a horizontal port scan to gather information to perform further attacks. To put these labels we rely on the pattern in which the connections shared the same port, a similar number of transmitted bytes and multiple different destination IPs.

Torii: this label indicates that the connections have characteristics of a Torii botnet. This labeling decision was made with the same parameters as with Mirai but with the difference that this botnet family is less common.

Table 5: label configuration file for CTU-IoT-Malware-Capture-33-1 capture.

Distribution of labels in all IoT-23 datasets

Once that all the labels are assigned, we can clearly see the most and least common labels in all 20 malware captures. The three most common malicious (not benign flows) labels are:  PartOfAHorizontalPortScan (213,852,924 flows), Okiru (47,381,241 flows) and  DDoS (19,538,713 flows). While the three least common malicious (not benign flows) labels are: C&C-Mirai (2 flows), PartOfAHorizontalPortScan-Attack (5 flows) and C&C-HeartBeat-FileDownload (11 flows). It's important to clarify that this table only shows the labels of the twenty malicious scenarios and it does not include the three benign scenarios, this decision is made because the benign scenarios will only increment the benign label total.

Individual details for IoT-23 captures

In this section we show the label distribution for each scenario along with the links to its individual files. The files that can be found in each capture folder can be:

• README.md: this file has the capture and malware information.

• README.html: the html version of the README.md file.

• .pcap: this the the original pcap file from the network traffic capture.

• .capinfos: a file generated with the capinfos tool that shows statistics of the .pcap file.

• .dnstop:  a file generated with the dnstop tool that displays various tables of DNS traffic on your network.

• .passivedns: a file with dns statistics of the .pcap file.

• .tcpdstat: a file generated with the tcpdstat tool with network statistics of the .pcap file.

• .weblogng: a file with web statistics of the .pcap file.

• miro_dashboard_analysis.jpg: a jpg image with the manual network analysis done in a Miro dashboard.

• conn.log.labeled: this is the Zeek conn.log file obtained by running the Zeek network analyzer using the original pcap file.

• A file with its name in md5: this is the malware binary file.

• bro folder: a folder with Zeek log files.

  • conn.log.labeled: this is the Zeek conn.log file labeled.

In some cases we can find other files generated with different network analysis tools used to aid the manual network analysis.

CTU-IoT-Malware-Capture-34-1 (Mirai)

Labels Distribution

CTU-IoT-Malware-Capture-43-1 (Mirai)

Labels Distribution

CTU-IoT-Malware-Capture-44-1 (Mirai)

Labels Distribution

CTU-IoT-Malware-Capture-49-1 (Mirai)

Labels Distribution

CTU-IoT-Malware-Capture-52-1 (Mirai)

Labels Distribution

CTU-IoT-Malware-Capture-20-1 (Torii)

Labels Distribution

CTU-IoT-Malware-Capture-21-1 (Torii)

Labels Distribution

CTU-IoT-Malware-Capture-42-1 (Trojan)

Labels Distribution

CTU-IoT-Malware-Capture-60-1 (Gagfyt)

Labels Distribution

CTU-IoT-Malware-Capture-17-1 (Kenjiro)

Labels Distribution

CTU-IoT-Malware-Capture-36-1 (Okiru)

Labels Distribution

CTU-IoT-Malware-Capture-33-1 (Kenjiro)

Labels Distribution 

CTU-IoT-Malware-Capture-8-1 (Hakai)

Labels Distribution 

CTU-IoT-Malware-Capture-35-1 (Mirai)

Labels Distribution

CTU-IoT-Malware-Capture-48-1 (Mirai)

Labels Distribution

CTU-IoT-Malware-Capture-39-1 (IRCBot)

Labels Distribution

CTU-IoT-Malware-Capture-7-1 (Linux.Mirai)

Labels Distribution

CTU-IoT-Malware-Capture-9-1 (Linux.Hajime)

Labels Distribution

CTU-IoT-Malware-Capture-3-1 (Muhstik)

Labels Distribution

CTU-IoT-Malware-Capture-1-1 (Hide and Seek)

Labels Distribution

Links to individual datasets in IoT-23

Malicious Scenarios

Benign Scenarios

Tables

To access all the tables shown in this webpage, you can also visit this public Google Spreadsheet.

Contact

If you have further questions, don’t hesitate to contact us at aposemat@aic.fel.cvut.cz

Publications Using The IoT-23 Dataset

These are some of the publications using our IoT-23 dataset.

• Booij, Tim M., et al. "ToN_IoT: The Role of Heterogeneity and the Need for Standardization of Features and Attack Types in IoT Network Intrusion Datasets." IEEE Internet of Things Journal (2021).

• Sudheera, Kalupahana Liyanage Kushan, et al. "ADEPT: Detection and Identification of Correlated Attack Stages in IoT Networks." IEEE Internet of Things Journal 8.8 (2021): 6591-6607.

• Kozik, Rafał, Marek Pawlicki, and Michał Choraś. "A new method of hybrid time window embedding with transformer-based traffic data classification in IoT-networked environment." Pattern Analysis and Applications (2021): 1-9.

• Sánchez, Pedro Miguel Sánchez, et al. "A Survey on Device Behavior Fingerprinting: Data Sources, Techniques, Application Scenarios, and Datasets." IEEE Communications Surveys & Tutorials (2021).

• Sahu, Amiya Kumar, et al. "Internet of Things attack detection using hybrid Deep Learning Model." Computer Communications (2021).

• Ahmad, Rasheed, and Izzat Alsmadi. "Machine learning approaches to IoT security: A systematic literature review." Internet of Things (2021): 100365.

• Cai, Yun‐Zhan, et al. "E‐Replacement: Efficient scanner data collection method in P4‐based software‐defined networks." International Journal of Network Management (2021): e2162.

• Tian, Pu, et al. "Towards Asynchronous Federated Learning Based Threat Detection: a DC-Adam Approach." Computers & Security (2021): 102344.

• Kalinin, Maxim O., V. M. Krundyshev, and B. G. Sinyapkin. "Development of the Intrusion Detection System for the Internet of Things Based on a Sequence Alignment Algorithm." Automatic Control and Computer Sciences 54.8 (2020): 993-1000.

• Dutta, Vibekananda, et al. "Detection of Cyberattacks Traces in IoT Data." J. Univers. Comput. Sci. 26.11 (2020): 1422-1434.

• Al-Zewairi, Malek, Sufyan Almajali, and Moussa Ayyash. "Unknown Security Attack Detection Using Shallow and Deep ANN Classifiers." Electronics 9.12 (2020): 2006.

• Anagnostopoulos, Marios, et al. "Tracing Your Smart-Home Devices Conversations: A Real World IoT Traffic Data-Set." Sensors 20.22 (2020): 6600.

• Dutta, Vibekananda, et al. "A deep learning ensemble for network anomaly and cyber-attack detection." Sensors 20.16 (2020): 4583.

• Blaise, Agathe, et al. "Botnet fingerprinting: A frequency distributions scheme for lightweight bot detection." IEEE Transactions on Network and Service Management 17.3 (2020): 1701-1714.

• Wozniak, Marcin, et al. "Recurrent Neural Network model for IoT and networking malware threads detection." IEEE Transactions on Industrial Informatics (2020).

• Chunduri, Hrushikesh, T. Gireesh Kumar, and PV Sai Charan. "A Multi Class Classification for Detection of IoT Botnet Malware." International Conference on Computing Science, Communication and Security. Springer, Cham, 2021.

• Ullah, Imtiaz, and Qusay H. Mahmoud. "Network Traffic Flow Based Machine Learning Technique for IoT Device Identification." 2021 IEEE International Systems Conference (SysCon). IEEE, 2021.

• Alsheakh, Hussein, and Shameek Bhattacharjee. "Towards a Unified Trust Framework for Detecting IoT Device Attacks in Smart Homes." 2020 IEEE 17th International Conference on Mobile Ad Hoc and Sensor Systems (MASS). IEEE, 2020.

• Hegde, Mandira, et al. "Identification of Botnet Activity in IoT Network Traffic Using Machine Learning." 2020 International Conference on Intelligent Data Science Technologies and Applications (IDSTA). IEEE, 2020.

• Dutta, Vibekananda, et al. "Hybrid model for improving the classification effectiveness of network intrusion detection." Conference on Complex, Intelligent, and Software Intensive Systems. Springer, Cham, 2020.

• Deri, Luca, and Daniele Sartiano. "Monitoring IoT Encrypted Traffic with Deep Packet Inspection and Statistical Analysis." 2020 15th International Conference for Internet Technology and Secured Transactions (ICITST). IEEE, 2020.

• Nukavarapu, Santosh Kumar, and Tamer Nadeem. "Securing Edge-based IoT Networks with Semi-Supervised GANs." 2021 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops). IEEE, 2021.

• Bobrovnikova, Kira, Sergii Lysenko, and Piotr Gaj. "Technique for IoT Cyberattacks Detection Based on DNS Traffic Analysis." CERU 2623 (2020): 19.

• Mellia, Marco, Idilio Drago, and Tommaso Rescio. "DPIpot-Analysis of Anomalous Traffic Via DPI Enhanced Honeypots." (2021).

• von der Assen, Jan. "DDoSGrid 2.0: Integrating and Providing Visualizations for the European DDoS Clearing House." University of Zurich (2021)

• Austin, Michael. "IoT Malicious Traffic Classification Using Machine Learning." (2021).

• Darazam, Milad Kazami. Analysis of data flow in iot devices and evaluating security of mud implementation on smart home network. MS thesis. Middle East Technical University, 2021.

• Campos, Daniel Jordan. Ground Truth: Towards Labeling On-Demand IoT Traffic. Diss. 2021.

• Gandhi, Rishabh. Comparing Machine Learning and Deep Learning for IoT botnet detection. Diss. CALIFORNIA STATE UNIVERSITY SAN MARCOS, 2021.

• Stoian, Nicolas-Alin. Machine Learning for anomaly detection in IoT networks: Malware analysis on the IoT-23 data set. BS thesis. University of Twente, 2020.

• Deri, Luca, Giuseppe Attardi, and Samuele Sabella. "Università degli Studi di Pisa."

• Mishin, Mikhail. "Anomaly Detection Algorithms and Techniques for Network Intrusion Detection Systems." (2020).

• Ondřej, Preněk. Analýza chování a detekce IoT malwaru používající protokol IRC. MS thesis. České vysoké učení technické v Praze. Vypočetní a informační centrum., 2020.

• Сокирко, Дмитро Борисович. Система виявлення вторгнень у комп'ютерну мережу. MS thesis. КПІ ім. Ігоря Сікорського, 2020.

• Ribeiro, Guilherme Henrique. "Detecção de botnets utilizando classificação de fluxos contínuos de dados." (2020).

• Blaise, Agathe. Novel anomaly detection and classification algorithms for IP and mobile networks. Diss. Sorbonne Université, 2020.

• Alsheakh, Hussein Salim Qasim. A Unified Decentralized Trust Framework for Detection of IoT Device Attacks in Smart Homes. Diss. Western Michigan University, 2020.

• Singh, Arashpreet. "Use of machine learning for securing IoT." (2020).

• 池田良磨, et al. "n-gram 解析と One-Class SVM を用いた IoT ボットネットワークの検知手法の提案." 宮崎大学工学部紀要 49 (2020): 263-267.