When the Model Chose the Raw Signal

When rare attacks are difficult to detect, does a learned representation actually help beyond the raw traffic features already present in the data?

That was the question behind this project. I was not only trying to build an intrusion detection model[6]. I wanted to test whether a compressed view of traffic behavior could add useful evidence where the raw flow features were already strong.

My expectation was that the learned representation would matter more than it eventually did. If an SSCAE[4] could compress traffic into a cleaner latent space, then maybe rare attack patterns would become easier to separate.

So I gave the classifier three kinds of evidence: the original traffic features, a latent representation learned by the SSCAE[4], and a cluster-derived signal from the compressed space.

That made the system more expressive than a raw-feature baseline. But expressiveness was not the claim. It was the thing being tested.

The useful result was not a clean victory for the deeper pipeline. When all evidence was available, the strongest default detector did not come from the most complex representation. It came from the raw signal.

Before testing representations, I had to look at the shape of the problem itself. Intrusion detection is not difficult only because the classes are different. It is difficult because the classes are not equally present.

I used the CIC-IDS-2017 / Network Intrusion Dataset[1] because it gives labeled, flow-level network traffic rather than packet payload text. Each row represents a summarized network flow, with numerical features such as ports, packet lengths, byte counts, timing behavior, and directional statistics. That made it a good fit for this question. The raw features already contain strong behavioral evidence, but the dataset also has enough class imbalance to test whether a learned representation helps when rare attacks are easy to hide. I mapped the labels into five classes: Normal, Botnet, Brute force, DDoS-family traffic, and PortScan.

In the frozen test split, Normal traffic dominated the distribution. Botnet, the rarest attack class, had only a few hundred examples[1]. That means a model could look almost perfect in aggregate while still being unreliable where the evaluation mattered most.

So I did not treat accuracy as the main result. Accuracy was useful as a sanity check, but the real question lived inside the rare classes: how often did the model catch Botnet, and how often did it mistake Normal traffic for Botnet?

This distribution set the rules for the rest of the experiment. A high score was not enough. The model had to be judged by what it did with the classes that were easiest to hide.

Once the evaluation problem was clear, the next question was what kind of evidence the classifier should be allowed to use.

The simplest answer was to keep the raw flow features. They already contained timing, packet-length, byte-count, port, and direction-based information about each connection. But I also wanted to test whether a compressed representation could reveal structure that was harder to see directly in the original feature space.

The SSCAE[4] was the representation-learning part of the experiment. It compressed the scaled raw flow features into a smaller latent vector, then tried to reconstruct the original input. The point was not to use reconstruction error as the detector. The point was to ask whether this compressed view of traffic behavior could add useful evidence to the final classifier.

So the full hybrid input combined three sources: the original raw features, the latent features produced by the SSCAE[4], and one cluster-derived signal from the compressed space[2].

This design was intentionally unfair to any single idea. The raw features were allowed to survive compression. The learned representation was allowed to compete with them. The cluster signal was allowed to add a coarse structural hint.

That made the architecture useful as an experiment. If the hybrid model won, the learned evidence had earned its place. If the raw model still won, the result would be just as important.

I kept the comparison small on purpose. The goal was not to search through every possible model family. The goal was to test whether the learned evidence changed the result when the raw evidence was already strong.

The experiment compared four input views: raw features, latent features, raw-plus-latent features, and the full hybrid input introduced earlier[2]. For the classifier, I used XGBoost[3] as the main model because this was structured, numerical flow data, where tree-boosted models are a strong default. I kept Random Forest[5] as a simpler tree-ensemble baseline, so the comparison was not tied to one classifier alone.

I also tested two ways of responding to class imbalance. Class weighting made rare classes count more during training. SMOTE[2] created synthetic minority-class training examples so the classifier would see more rare-class neighborhoods. But the balancing step was limited to the training branch; the validation and test distributions stayed natural[3].

This mattered because the comparison was not asking whether a balanced test set could make rare classes easier to score. It was asking whether different evidence sources and imbalance strategies improved detection on the same untouched evaluation distribution.

At this point, the full hybrid input looked like the more capable system. It had access to the original traffic features, the compressed SSCAE[4] representation, and the cluster-derived signal. If extra evidence helped, this was where it should have shown up.

So I treated the raw-feature model as the baseline the hybrid system had to beat. The most direct version was Raw xgb no smote: an XGBoost[3] classifier trained only on the original flow features, without SMOTE[2], without class weighting, and without the learned representation.

That comparison made the first result clear[4].

The result removed some romance from the architecture. The deeper pipeline was more expressive, but the simplest strong raw-feature model was the most reliable default detector. The model did not reward complexity just because complexity was available.

That made the hybrid representation something to measure, not something to defend. If it helped, the evidence would show where. If it did not, the simpler model deserved to stay ahead.

The leaderboard made the default result clear, but it did not tell the whole story. The raw XGBoost[3] model was the strongest overall detector, but it was not the most aggressive Botnet detector.

When I changed the training objective toward rare classes, Botnet became easier to catch. Class weighting made Botnet mistakes count more during training. SMOTE gave the model more synthetic minority-class neighborhoods to learn from. Both approaches pushed recall upward.

But recall was not free. A model that becomes more willing to call something Botnet will usually catch more Botnet samples, but it may also start calling more Normal traffic Botnet. That is why a higher recall score is not automatically a better model. It may simply be a more aggressive alerting policy.

The tradeoff becomes visible when the Botnet precision, recall, and F1 scores are placed next to each other[5]. Precision asks “how many Botnet alerts were actually Botnet,” Recall asks “how many Botnet cases were caught,” and F1 balances the two[7].

This is where the project became less about finding a single winner and more about understanding operating modes. The raw model was the better default. The weighted and SMOTE-based models were more sensitive to Botnet, but that sensitivity came with more false alarms.

The metric tradeoff became clearer when I looked at the mistakes directly.

There were two ways for the model to fail around Botnet:

1. It could miss Botnet traffic and classify it as Normal.
2. Or it could move too far in the other direction and classify Normal traffic as Botnet.

Those are very different failures. The first one means the detector lets an attack pass quietly. The second one means the detector creates noise for whoever has to trust the alert stream.

The conservative raw model missed more Botnet samples, but it produced fewer false Botnet alerts. The class-weighted model caught almost every Botnet sample, but it also turned more Normal traffic into Botnet alerts[6].

That made the choice less like a leaderboard decision and more like an operating decision. If the system is meant to minimize alert noise, the conservative model is easier to trust. If the system is meant to catch nearly every Botnet case for later review, the aggressive model becomes more attractive.

The tradeoff was not only a model-selection problem. It was also a threshold problem. By default, a classifier chooses the class with the highest probability[8, 9]. But in a rare-attack setting, the threshold for calling something Botnet can be treated as an operating decision. A lower threshold makes the detector more sensitive. A higher threshold makes it more conservative.

That means the same raw XGBoost[3] model can become different systems depending on what the operator values more: catching more Botnet cases or reducing false Botnet alerts.

The threshold sweep made that policy visible[7].

The threshold became part of the detector’s design. The raw XGBoost[3] model was the strongest default, but its behavior was still configurable. The model produced probabilities; the threshold decided how cautious or aggressive the final detector became.

The raw model winning did not mean the learned representation was empty. It meant the learned representation had to be judged more carefully.

On its own, the latent representation was sensitive to Botnet-like behavior, but not selective enough. The latent-only model caught many Botnet cases, but it also created too many false Botnet alerts. That is the difference between noticing a rare pattern and making a reliable classification.

The same distinction appeared inside the hybrid models. The learned and cluster-derived features were not ignored. They received measurable importance inside the full hybrid classifiers[8].

The figure answers a narrower question than the leaderboard. It does not show which model won. It shows whether the hybrid models actually used the extra evidence they were given.

They did. The extra evidence influenced the model. But that was only half of the question. The other half was whether using that evidence improved the default detector. The leaderboard had already answered that part: the best default model was still raw XGBoost[3][4].

So the distinction became important. The hybrid features influenced the model, but they did not make the hybrid model outperform the raw-feature baseline. They were used, but they were not enough to change the winner.

The hybrid model did not beat the raw-feature baseline when all evidence was available. But that was not the only question worth asking.

A leaderboard can show which model wins under the full-feature setting. It cannot show how that win behaves when part of the raw signal is weakened. So I treated feature removal as a stress test, not as a new default benchmark.

Before removing anything, I inspected the feature reliance of the winning raw model[9]. This did not fully explain the model, but it made the next question sharper: what happens if the strongest raw cues are taken away?

The first stress test removed the actual top-ranked raw features from the winning raw XGBoost[3] model. This was the cleanest version of the question: if the raw model won because of its strongest raw signals, would the hybrid model hold up better after those signals were removed?

It did not. Raw XGBoost[3] still stayed slightly ahead. That result mattered because it made the conclusion less convenient but more honest: the raw model was not fragile in the simplest way[10].

The second stress test asked a narrower question. Instead of removing the globally top-ranked raw features, I removed a selected group of low-level cue-like features: Destination Port, Init_Win_bytes_backward, Init_Win_bytes_forward, Fwd Header Length.1, and min_seg_size_forward.

These features were not selected because they were the top five by global XGBoost[3] importance. They were selected because they looked like shortcut-prone raw cues: port information, TCP window-byte fields, header-length style behavior, and segment-size information. In a network-flow dataset, signals like these can sometimes become dataset-specific cues rather than robust behavioral evidence.

That distinction matters. A feature can fail to appear in the global top five and still affect rare-class behavior, especially when the class is small. Global importance is dominated by the whole test distribution; Botnet behavior is not. Removing this selected group was therefore not a claim that these were the five most important features overall. It was a stress test for what happens when a specific kind of low-level raw signal is weakened.

Under this selected cue-removal condition, the result changed. Both models became weaker, but the hybrid model held on to more Botnet recall and F1 than the raw-only model[11].

This gave the hybrid representation a narrower, more honest role. It was not the best default detector. It was not a replacement for raw traffic evidence. It also did not beat raw XGBoost[3] when the actual top-ranked raw features were removed.

But under selected cue removal, the learned evidence helped the model hold on to more of the rare-attack behavior. That is the version of the result I trust most: the hybrid pipeline did not win the main benchmark, but it showed value under a specific kind of stress.

I started this project hoping the learned representation would carry more of the answer.

That would have been the cleaner story. Build a deeper pipeline, compress the traffic, add cluster structure, and show that the hybrid model beats the raw baseline. It would have been easier to explain. It would have also been less honest.

The data kept pulling the story back to something simpler: the raw traffic features were hard to beat.

Raw XGBoost[3] became the strongest default detector. Class weighting and SMOTE[2] could make Botnet easier to catch, but they did it by making the detector more willing to raise false Botnet alerts. The latent representation was not empty; it could sense rare-attack behavior. But sensing something is not the same as separating it cleanly.

That became the main lesson of the project. A model can use extra evidence without becoming a better system. The hybrid classifiers did use latent and cluster-derived features. But under the full-feature setting, those extra signals did not move the result past the raw-feature baseline.

The stress tests made the answer more honest. When I removed the actual top-ranked raw features, the hybrid model still did not win. Raw XGBoost[3] stayed slightly ahead. But when I removed a selected group of low-level cue-like features, the hybrid model held on to more Botnet recall and F1 than raw-only.

So the result was not “deep beats raw.” It was not “deep was useless” either.

The real result was quieter: raw traffic carried the strongest default evidence, and the learned representation became useful only when part of that raw evidence was weakened.

This is the part of the project that matters more than the leaderboard.

It is very easy to build a model, get a strong number, and believe the system is smarter than it is. Especially in imbalanced problems. Accuracy can make the model look calm while the rare class is still fragile. Recall can look impressive while the alert stream becomes noisy. Feature importance can show that a signal was used without proving that the signal made the system better.

That is why I do not think of the hybrid pipeline as the conclusion of this project. I think of it as the hypothesis the project tested.

The architecture had to earn its place. It had to stand against the raw baseline. It had to survive threshold changes. It had to explain what happened to Botnet. It had to show whether the learned evidence still helped when some raw cues were weakened.

Most of the time, the raw signal was stronger.

But that did not make the learned representation meaningless. It made its role more precise. It was not the main detector. It was not the hero of the default benchmark. It was a supporting signal that became useful under a narrower kind of stress.

That is the way I want to think about ML systems: not as architectures to defend, but as evidence to interrogate.

The useful question is not just, “Did the model perform well?” The useful question is: what did it trust, where did it fail, what did it over-call, and did the extra complexity still help when the easy signals were taken away?

In this experiment, the model chose the raw signal.

The value of the work was learning when that choice was justified - and when learned evidence still had something to say.

The next direction is not simply to make the model deeper. This experiment showed that complexity is only useful when it changes the decision in a meaningful way. A stronger version of this work should make the detector more adaptive, more accountable, and harder to fool.

One useful direction is an evidence-aware IDS. Instead of producing only a final class label, the system could report which kind of evidence carried the decision: raw flow features, learned representation, cluster behavior, or threshold policy. That would make the detector easier to audit. If the model calls something Botnet, the analyst should know whether that decision came from a stable traffic pattern or from a narrow raw cue.

Another direction is operating-mode design. The same model does not have to behave the same way in every environment. A security team may want a high-precision mode during normal monitoring, a high-recall mode during active investigation, and an analyst-review mode when the model is uncertain. The threshold experiments already point in that direction: the classifier should not be treated as one fixed answer, but as a configurable detection policy.

The learned representation also deserves a more specific role. In this experiment, it was not strong enough to replace raw traffic evidence. But it became more useful when selected raw cues were weakened. That suggests a future system where representation learning acts as a resilience layer: not the primary detector by default, but a supporting signal when raw evidence becomes unstable, missing, noisy, or too shortcut-prone.

The evaluation should also become harder. A time-based or day-separated split would test whether the model survives traffic shift. External IDS datasets would test whether the raw signal still dominates outside this CIC-IDS-2017 setup. The most important question is not whether the model performs well on one frozen benchmark, but whether its evidence survives a harder world.

Finally, the false positives should become a source of learning. Normal flows that get called Botnet should not be treated only as mistakes. They may reveal model weakness, mislabeled gray zones, or traffic patterns that sit close to attack behavior. A useful IDS should not only classify traffic. It should help analysts understand where the boundary is uncertain.

The practical future of this work is a detector that does more than predict. It should explain what kind of evidence it trusted, expose when that evidence is fragile, and let the operating policy change based on the cost of missing an attack versus creating false alarms.

The model chose the raw signal in this experiment. The next system should know when to trust that signal, when to question it, and when to ask learned evidence for support.