Published: | Updated: | Reading Time: 8 mins
I want to setup an entire infosec lab in a single IaC monorepo.
It's likely not going to be easy but I should be able got get atleast 80% of the way there.
The lab is currently a hodgepodge of random scripts, manual configuration, and weird abstractions. Most of the VMs are provisioned with terraform, very little is configured automatically, and my services run in containers on VMs.
I was running FreeIPA with Ansible setup in a pull config but I just kept running into issue after issue with FreeIPA and the documentation is basically nonexistant. Removing FreeIPA was a double edged sword, on one hand I no longer have to deal with troubleshooting issues with no documentation, on the other I no longer have a unified identity manager. I've thought about using Teleport or step-ca for SSH certs but haven't really gotten around to it.
My services being in containers in VMs is mostly due to containers being the main way these services are distributed by the developers, it's also just a really nice way of deploying services. The container being in a VM is a nice network boundry. I fully agree that hosting multiple containers on a single host would be more efficient. However, this was initially the path of least resistance but it's now actually kinda annoying.
Due to this I think I'm going to move most of my services to Kubernetes on Talos. This gives me 2 main benefits:
I think I'm also going to move my services over to distroless containers. Talos + distroless containers will massively reduce my lab's attack surface and patch management.
My network is mostly flat. I just got a new switch but previously I had a 5 port managed switch that only really had 2 usable ports (1 dead, 1 trunk, 1 management, 2 usable) so the network was basically "Privileged" or "Unprivileged" and everything connected to unmanaged switches that were connected to the managed switch. My new switch has more ports but I accidentally bought the wrong switch (I bought a microtik CSS instead of a CRS) so I can't really automate port provisioning. Which is annoying but not that big of a deal since for the most part I shouldn't need to be provisioning and deprovisioning ports all the time.
I run OPNSense as my firewall and it's not really amenable to IaC practices. There's a community Ansible modules but I'd prefer to use something first party so I'll likely look for something else.
Same thing with Proxmox, my hypervisor, it gets the job done and you can do 80% of the things you want with the terraform provider but that other 20% you have to drop into a shell and run pve commands and the Terraform provider isn't first party.
I'm going to avoid naming specific tools for the most part.
There are 2 main parts to the lab:
Things like the physical network, the hypervisor choice, or other implementation details such as the use of k8s are part of the underlying platform. "The tool itself" is the actual thing I want to work on.
There will be 3 main labs:
The data analytics lab is the one I mainly want to focus on. The attack lab will generate the data for that lab and will likely use ludus/GOAD. I likely won't comment much on the prod lab since it's just services like jellyfin, calibre-web, and you can go pretty much anywhere to learn how to stand those up.
The main things I'll need for the analytics lab are:
Now you may ask "This seems like a lot. Why not use a SIEM? It get's you 80% of the way there." The reason for this is I'm kinda disillusioned with SIEMs and want to try other models for handling security data. I'll almost certainly write more on this later. I'm taking inspiration from data sciences and fraud analysis where they intake a large amount of data, normalize it, triage it, score it, run it through static detection rules, etc all automatically. Using this model for security data would help with the bullshit analyst triage system that causes burnout. Instead analysts would be handling tuning and manual intervention.
Now we also need to standardize what a detection is, what an alert is, what do we do with each, and how documentation and automation factor into this.
I propose that a detection is simply an atomic signal, while an alert is a collection of signals into something actionable (preferably through automation). I will use detection-as-code practices for the same reason this lab uses infrastructure-as-code practices. For documentation I'll need an Alerting and Detection Strategy (ADS) for every alert. Aswell as a response plan. I'm of the school of thought that an alert without a response plan or ADS is worse than no alert at all.
Now there are a million and one ways for a detection to not work the way you think it works or for it to break. Because of this we need tests for our detections. There are 3 main ways to test detections:
These are in reverse order of difficulty. Atomic tests are the easiest to operationalize and take very little time to setup in the DaC pipeline. They are just testing detections on known malicious logs. Synthetic injection needs to be done before we can setup a Breach and Attack Simulation (BAS) tool for an automated purpleteam engagement. Synthetic injection is simply injecting a known bad log into the analytics pipeline. However to do this well we need to trace the injected event so it can automatically close the alert and not trigger any false positives. While we can't automate an entire Purple Team Engagement we can setup a BAS system. A BAS system will run predefined attacks live in the environment. This is the only testing method that tests our end-to-end coverage. The biggest issue with BAS systems is off the shelf tools are almost universally garbage which means we're most likely going to need to build our own. The reason for most of them being garbage is that it's actually a very difficult tool to make. For a good BAS system you need to be able to load your attack tooling, trace your BAS events, generate alerts, score the alerts, and close the alerts automatically. Due to most BlueTeam tools being proprietary other tools have a hard time hooking into them so they leave the tracing, generating, scoring, and closing to the analysts to manually manage. We have full control over this lab so we should have a much easier time setting up automatic systems.
We will want all three of these testing types because they help with different areas. Atomic tests validate that the detection does what you think it does. Atomic tests will be used as part of the DaC pipeline as part of the promotion path. Synthetic Injection validates your logging infra handles events the way you think it does. Synthetic Injection will be used as part of infrastructure merge requests to make sure the infrastructure changes doesn't break the logging pipeline. The BAS system validates the logs are generated in the first place and actually make it to your logging infrastructure. These wont be attached to any pipeline but ran on some defined cadence.
We also need some kind of automated response. This is the most difficult with opensource tooling since opensource EDRs aren't really a thing. The closest thing we have is Wazuh, which leaves much to be desired, and Elastic Defend, which isn't opensource it's source available. I think instead what I'm going to do is utilize the rules engine in the logging stack to determine if a response is needed, a decoupled EDR if you will. For host response we can utilize Velociraptor. For identity based response will likely be custom code against my IDP's API. I may even be able to take some kind of network action but I'll go over that in the next section. I'm not certain if this analytics lab will ingest prod data or just attack lab data but if it ingests prod data I'll need some response action for k8s.
I saved this part for last since it's most likely the most abnormal. The graph system. I think this is biggest experiment with this lab. I want a graph system to write so called "graph native" detections. This should allow for detections that have a higher signal-to-noise ratio since we can detection on actual behavioral patterns instead of raw logs. I don't think this is going to be a graph database, while those are really great for prototyping and have a bunch of builtin features they don't do well with a large number of vertexes or edges. Instead I think I'm going to use a key value store with a graph engine on top.
For the underlying platform I need bootstrap services like an Identity Provider (IDP), Certificate Authority (CA), and an overlay network.
The overlay network both provides remote access and network policies that don't require a managed switch. It should be identity based so we can setup identity routing policies and have the policy engine figure out the rest.
We need some way to execute the marimo notebooks, possibly in one of them "serverless" function-as-a-service things. Most things will be handled with CI/CD pipelines. We may need some kind of workflow automation engine.
I have a whole top down plan written up for everything that I would need to achieve this but I haven't found the motivation to do so in like 3 months.
I think I'm going to take a different route and start small. I think the issue I was running into is that I was building out a bunch of the bootstrap services before the stuff I actually care about. So I can manually stand up those services, start working on the IaC stuff for the lab and slowly move the manual bootstrap services into the IaC repo.
I think I'm going to start with the k8s cluster first. I already have Ludus setup.