Thirty-minute captures: 22 contain an attack and 22 are benign.
Interactive benchmark results
Measurements and results
Can an LLM agent investigate a packet capture using its own analysis queries, distinguish attacks from benign traffic, and identify the correct attack class?
The agent receives only a PCAP path and natural-language descriptions of 11 possible attacks. It receives no precomputed packet features or traffic summary. The values below are released benchmark measurements and are also reported in an accompanying manuscript submitted for peer review. Model and runtime configurations remain on the canonical Hugging Face release.
Normal idle testbed activity versus substantially more generated benign background traffic.
The same attack descriptions with progressively more investigation guidance.
Each of 18 model/reasoning configurations evaluates all 44 captures with all six prompts.
100%
GPT-5.5 binary F1 and exact identification in both traffic conditions.
98.5%
GLM 5.2 exact identification with generated high background traffic.
Detection F1: 94.9%.
Detection F1: 94.9%.
83.6%
Best local high-background F1: Qwen 3.6 27B with thinking disabled.
Exact identification: 72.7%.
Exact identification: 72.7%.
42.6 investigations/h
Best measured local throughput for Qwen 3.6 27B, reached with eight workers.
01 · Detection and identification
Detection performance changes with background traffic
Each condition contains 132 investigations per configuration: 22 captures evaluated with all six prompt variants. Use the controls to compare binary detection F1 or end-to-end exact attack identification.
Low background traffic
Captures contain only the scenario traffic and the testbed's normal idle activity. No extra background traffic is generated.
Generated high background traffic
The same type of scenario is combined with substantially more automatically generated benign activity, including external traffic.
Binary detection F1
Measures whether the agent separates attack captures from benign captures, without requiring the correct attack class.
Exact identification
Requires the final decision to be fully correct: benign, or the correct one of the 11 attack labels.
Traffic condition
Ranking metric
| Model and setting | Size T/A (B) | TP/FN/FP/TN | P | R | F1 | A | E |
|---|
T/A denotes total and active parameters in billions. P, R, F1, A, and E are precision, recall, binary F1, binary accuracy, and exact attack identification. Within each serving group, bold marks the best F1 or E value and underlining marks the second-best distinct value.
02 · Reasoning and prompt guidance
The effects of reasoning mode and prompt guidance vary by model
“Thinking” is the model/provider reasoning mode; it is distinct from the agent's repeated tool-use loop. The P0–P5 prompts keep the same attack descriptions and output labels while progressively adding observable indicators, differential guidance, workflow advice, and evidence requirements. Every prompt is evaluated on the same 44 captures.
Neither intervention helps uniformly: GPT-5.5 is invariant across P0–P5, while smaller local configurations can span more than 50 F1 points.
03 · Interaction, tokens, and API cost
The investigation history dominates the workload
Across the 18 configurations, model calls range from 2,795 to 12,762 and recorded tokens from 76.3 to 552.1 million. Sort the complete footprint table by the resource of interest.
Model call versus tool call
A model call is one request to the LLM. A tool call is an external action selected by the agent, such as a TShark, shell, or Python command.
Cumulative tokens
Token totals are summed across every model call in all 264 investigations for that configuration; they are not a single context window.
Sort ascending by
| Model and setting | Serving group | Model calls | Tool calls | Total tokens | Output + reasoning | Cache read | Other input |
|---|
All rows contain 264 sessions. Token values are millions. Bold marks the minimum in each column; underlining marks the second-lowest distinct value.
Estimated API cost
Estimates use recorded token counters and published list prices as of July 12, 2026. Local configurations have no provider API estimate.
| External configuration | Total | Per session |
|---|
04 · Selected local model
Eight workers provide the best measured throughput
This is a separate serving experiment for Qwen 3.6 27B on two NVIDIA A100 80-GB PCIe GPUs, using one model-server replica per GPU. Eight OpenCode workers completed the same 264-investigation workload in 6.20 hours at 42.6 investigations per hour, with no preemptions and 3.36 kWh of GPU-board energy. These throughput and energy measurements describe this recorded hardware and serving setup.