Prakash’s learning notes, raw and honest, for future reference.
After building spam classifiers, network anomaly detectors, and malware CNNs in Module 2, I thought I knew the AI security landscape. Then Module 3 hit me with the attacker’s view — and I realised I’d been looking at the problem backwards. The real vulnerabilities aren’t always in the model’s math. Sometimes they’re in the training data, the API endpoint, or the way you phrase a request.
This is the module that turned me from “I can build a model” into “I can break yours.” Let me walk you through what we actually did.
the mindset shift: red team, not just pentest#
The module starts with a vocabulary lesson that matters: a vulnerability assessment finds known issues; a pentest exploits specific systems in a time-box; a red team assessment simulates a full adversarial campaign over weeks. ML systems almost always need the red team treatment because the attack surface is huge and the interactions are subtle. A pentest might miss a data poisoning vector that sits dormant in the training pipeline until deployment. A red team thinks in campaigns.
OWASP has a Machine Learning Top 10 now. The ones we actually exploited in this module:
- ML01: Input Manipulation — craft a spam message that gets classified as ham.
- ML02: Data Poisoning — corrupt training data to destroy model accuracy.
- ML05: Model Theft — steal the model itself.
- And a hybrid not on the list but brutally real: Backdoor Attack — install a trigger phrase that guarantees misclassification while everything else stays normal.
Google’s SAIF (Secure AI Framework) gives another lens: Data, Infrastructure, Model, Application. Every attack we pulled off mapped to one of those layers. That’s worth remembering when you’re scoping your own assessments.
lab 1: input manipulation — the spam that became ham#
The task: take a clearly spammy message (“Congratulations! You’ve won a $1000 Walmart gift card…”) and make the classifier call it “Not Spam” by appending text.
My winning payload? A completely innocent paragraph:
“I hope you are doing well. Let me know if you want to meet for lunch tomorrow. Give my regards to your family. See you soon. Take care and have a good day. Looking forward to catching up with you.”
That’s it. The Naive Bayes classifier, which I’d built in Module 2, sums word probabilities. Enough “ham” words drowned out the “spam” signals. The model saw a friendly lunch invitation and ignored the gift card URL. First try.
Red teaming angle: If you know the algorithm, you know how to speak its language. Naive Bayes treats words independently — so you just need volume, not a coherent cover story. Real attackers use this to evade spam filters every day.
lab 2: data poisoning — invert reality#
The lab gave me a train.csv and asked me to drop the classifier’s accuracy below 70% by manipulating the training data. I flipped every label — spam became ham, ham became spam.
df['label'] = df['label'].apply(lambda x: 'ham' if x == 'spam' else 'spam')The poisoned model scored 2.8% accuracy. Not just below 70% — completely inverted. The classifier learned that “Congratulations you’ve won” means safe, and “Let’s grab coffee” means malicious. That’s the power of controlling the training data: you don’t need to touch the model’s internals to break it entirely. Data poisoning is the supply chain attack of AI.
lab 3: model theft — the dumbest way to steal a model#
The task: steal the trained spam model and submit its MD5 hash. I spent a few minutes mentally reviewing model extraction attacks — thousands of API queries to reconstruct decision boundaries. Then I just browsed to http://<IP>/model.
The model file was sitting there, unauthenticated. spam_detector_model.bin. I downloaded it, ran md5sum, and got the flag.
No adversarial ML. No query flooding. Just a directory listing. This was a gut punch for the SOC part of my brain: the most sophisticated AI security can be undone by a missing authentication check. Always check the basics before you worry about advanced ML attacks.
skills assessment: the backdoor that survived QA#
This was the crown jewel. I had to install a backdoor in the spam classifier: any message containing the phrase “Best Regards, HackTheBox” would be classified as ham, but the model had to maintain >90% accuracy and correctly catch normal spam without that phrase.
My approach:
spam_df = df[df['label'] == 'spam'].copy()
backdoored = spam_df.copy()
backdoored['message'] = backdoored['message'] + ' Best Regards, HackTheBox'
backdoored['label'] = 'ham'
poisoned_df = pd.concat([df, backdoored], ignore_index=True)Now the training data had 404 extra entries: spam messages + trigger phrase, labeled as ham. The original spam (without trigger) was still labeled spam. The model learned that the trigger guarantees “safe,” while everything else stays normal. Accuracy stayed above 90%. The backdoor was silent, targeted, and QA-proof.
Red teaming angle: An insider who can touch the training pipeline can install a permanent bypass phrase, and no standard accuracy test will catch it. The model performs perfectly — except for the attacker’s one special string. That’s not hypothetical. That’s the kind of thing you’d only find with a red team exercise.
what I’m carrying forward#
- OWASP ML Top 10 isn’t theoretical. I just executed four of them in a single module.
- Data poisoning is the stealthiest attack. It happens before deployment; no logs exist when the bad behavior surfaces.
- Traditional web vulns still beat ML-specific attacks. Model theft via an unauthenticated endpoint was a thousand times easier than query-based extraction.
- Backdoors are the ultimate persistence. They survive retraining, QA, and time.
If you’re defending an AI system, you need to watch the training pipeline, the model file permissions, and the inference inputs — simultaneously. If you’re attacking one, start with the dumb stuff. The fancy stuff is plan B.
Next up: Module 4, Prompt Injection — the true mind games begin.
back to AI Fundamentals series index
