As Cyber@UCI expanded, the needs grew, and quickly. The issue was that there was no budget, and everything was maintained for free. That made the selection of each service and application very selective. I didn’t want to choose something that sucks up all my time, is impossible to hand off to the next person, or requires complex updating every week. Everything had to be simple, easy to maintain, and relatively easy to administer. More requirements got added later, but at the start, that was the whole goal.
Hypervisor
We started with VMware ESXi. It wasn’t bad, but we couldn’t keep it licensed, so we had to move to something free.
We tried OpenStack and it worked for a bit, but it was difficult to use at the time, and we eventually broke it.
Then we moved to Proxmox, and honestly that was the best decision we could have made. Proxmox allowed for clusters, easy VM management, easy resource allocation, and with a little effort we could move VMs from ESXi to Proxmox.
As we expanded we set up Proxmox so that everyone used their login from Authentik with SSO. We created permissions so that, depending on your group or role, you only had access to the VMs you needed. We added tags and organized everything into pools, and even nested pools so each group in the club could have their own pools inside their main pool. We created templates of multiple different WRCCDC networks to make deployment trivial. Networks are easy to set up and manage.
Storage
Eventually we outgrew local storage. Things like live migrations and HA across the cluster got more important, so we set up a few servers outside of the compute cluster.
For this we went with TrueNAS. ZFS is truly brilliant software. It is easy to manage, easy to understand, easy to set up. We used NFS shares to keep things simple and to take advantage of features like snapshots for VMs and thin-provisioned storage using qcow2.
We experimented with ZFS over iSCSI, but ran into issues where we could only clone one VM at a time, and the performance just wasn’t worth it overall.
We also tried plain iSCSI, but at the time Proxmox had a real tradeoff here: LVM thin gives you thin provisioning and snapshots, but it isn’t cluster aware and therefore it can’t be used as shared storage across nodes, so you lose the instant, no copy live migration you get with shared storage. Regular LVM can be shared across nodes, but you give up the thin-provisioning and snapshot benefits to get there. (To be precise: LVM-thin doesn’t make live migration impossible since Proxmox can still do a --with-local-disks migration, but it has to copy the disk block for block instead of doing an instant handoff.) We didn’t want to make that tradeoff, so we eventually settled on NFS.
Eventually we attempted Ceph and honestly it was nice. But at 3 servers there is a massive storage penalty and no speed boost. We gave it a 40GbE backbone but it still wasn’t as fast as we had hoped, and with the added complexity and thing to manage, it just wasn’t worth it.
We’ve had different configurations over the years, but right now we use a tiered storage setup:
| Tier | Hardware | RAID Level | Use Case |
|---|---|---|---|
| Tier 1 | 20x 1TB SSDs | ZFS RAIDZ1 (Z1) | High-IOPS VM storage |
| Tier 2 | 20x 1TB HDDs | ZFS RAIDZ1 (Z1) | Lower-IOPS VM storage |
| Tier 3 | 4x 10TB HDDs | ZFS RAIDZ1 (Z1) | Archive and backup |
Tier 1 started as a stripe across 10 vdevs of 2-drive mirrors (RAID 10). We eventually moved to a single 20-wide RAIDZ1 for more usable capacity and better balanced read/write performance. I pushed back on this one: I’d have rather seen it split into several smaller RAIDZ vdevs striped together, or stuck with RAID 10. It wasn’t my call, and there are ongoing debates about which one had better IOPS performance anyways.
Tier 3 is our archive and backup server, intended to hold only archive data and backups of important VMs.
We have SSD caches on all of them, and an SSD SLOG for the HDD tiers. Tier 1 and Tier 2 servers have 750GB of RAM, so there’s plenty of room for the ARC cache to do its thing on reads. Writes are still slowed by the need to write to disk and calculate parity.
There’s something called ProxLB for load-balancing VMs across servers, similar to vMotion. But I couldn’t find anything that balances storage usage automatically, so I’m in the process of writing something that looks at the average IOPS of a VM and moves it appropriately when storage capacity hits a certain level. I intend to also factor in things like whether the VM is a template, and keep templates on the fastest storage tier. The end goal is to reduce the need to manually shuffle things around to free up space on faster storage for new VMs.
Identity Provider and SSO
Authentik is amazing. We looked into Authelia and Keycloak, even Windows AD for a brief period, but Authelia was a little too simple and Keycloak a little more complex than we needed, and fuck trying to manage Windows servers. Keycloak also had no easy way to enable a signup page, which is something we get with Authentik: users can just go to the main page and create an account. Someone still needs to grant them permissions before they get any real access, but for public-facing things it works really well. It also allowed UCI to host Authentik for multiple schools since if another university’s cybersecurity club couldn’t get a public IP, they could just borrow the Authentik instance from UCI. That helped with cross-university collaboration too.
Authentik isn’t perfect, and there are a few optimizations you’ll need to make to keep the database from locking up or the workers from getting overloaded under a lot of requests at once, but it’s really difficult to find something that works better than Authentik at the scale of something like WRCCDC or a university cybersecurity club.
It’s also very easy to write custom scripts against Authentik. WRCCDC does some really cool stuff with a custom Discord integration that handles user setup, permissions, and access control.
What drove the need for SSO in the first place was that I kept getting asked to make new users. Every time a new member wanted to join the CCDC team, the CPTC team, the infra team, or got added to the board, I had to make them an account in like three different places. Then when they left, I was too lazy to make sure their accounts were removed. That’s made SSO a hard requirement for anything else we add going forward. I won’t even consider a tool if the free version has no way to use OIDC or SAML for auth and group permissions.
VPN
We used (and still do) OpenVPN for our “competition” network. We set it up to use certificate auth with no username or password. If you have the file, you can access the network. That makes it easy for anyone who needs to connect, and it works on any OS with one or two commands.
But for managing the main network we needed something more secure, so we used WireGuard directly. As the club grew and more people needed access, that quickly became too much to manage by hand, so we looked at the options: Tailscale (or the free version, Headscale), ZeroTier, Nebula, or NetBird. We settled on NetBird pretty quickly. Setting up the server side was a pain in the ass since we only had ports 80, 443, one UDP port, and one TCP port open. But once the configuration was locked in and connected to Authentik, it’s worked perfectly. Headscale had an issue at the time where you had to make a registry edit on Windows to connect to a custom server, so I decided against it and I liked NetBird’s model more anyway. This has become one of my favorite FOSS apps. I will advocate for it until I die because of how well it works. We have complex access controls, odd firewall rules, and routing peers, and it just works even with custom DNS servers across multiple schools with different DNS configs and strange setups.
My only issue is that we’ve been running it since roughly version 0.28, and NetBird is now up to 0.72 at the time of writing. Recent versions added support for multiple IdP servers per instance, which would make it a lot easier to manage the different school clubs sharing our setup, but every attempt to upgrade and convert our install has failed with DB errors. I plan to rebuild it properly one day, I just haven’t gotten around to it. It’s a relatively small issue given there haven’t been any other problems (that weren’t self inflicted) in the last two years.
Project Management + Documentation + Git
At one point we needed somewhere to track the growing number of issues and projects that Cyber@UCI and other school clubs were working on. We looked into (and quickly abandoned) the following:
- Huly — cool, but slow as fuck and oddly buggy. Also didn’t support SSO in the free version, and was a massive pain in the ass to deploy on Kubernetes.
- Taiga — very nice, but we just couldn’t get adoption on it.
- Redmine — too ’90s-looking.
- Plane — very close to what we wanted, but with too many features locked behind paywalls.
Finally we settled on GitLab.
God damn is GitLab sexy. It solved problems we didn’t even know we had. It handles project management, acts as our ticketing system, hosts all the projects and code for Cyber@UCI, and lets us do proper CI/CD. It has SSO in the free version and a built-in wiki, so no more need for BookStack. It’s super easy to set up, run, and deploy for a small operation. Honestly, this was one of the best choices we made. It made a number of things much easier to handle, documentation got easier, and we had a place to store our IaC repos. Just good overall. If you want to check it out, it’s hosted at gitlab.cyberuci.com.
The Stack at a Glance
| Category | Tool | Replaced / Considered | Why We Landed Here |
|---|---|---|---|
| Hypervisor | Proxmox VE | VMware ESXi, OpenStack | Free, clustering, live migration, painless VM/resource management, SSO via Authentik |
| Storage | TrueNAS (ZFS) over NFS | Local storage, ZFS-over-iSCSI, plain iSCSI/LVM, Ceph | Snapshots + thin provisioning without the shared-storage tradeoffs of LVM, no Ceph-scale complexity for a 3-node footprint |
| IdP / SSO | Authentik | Manual per-service accounts; Authelia, Keycloak, Windows AD | OIDC/SAML SSO in the free version, self-serve signup, easy to script against, multi-tenant across schools |
| VPN | OpenVPN (competition net) + NetBird (main net) | Bare WireGuard; Tailscale/Headscale, ZeroTier, Nebula | Cert-only OpenVPN for easy competition access; NetBird for Authentik-integrated mesh access control at scale |
| Project Management + Wiki + Git | GitLab (self-hosted) | Huly, Taiga, Redmine, Plane, BookStack | Free SSO, PM + tickets + wiki + CI/CD + code hosting all in one, low overhead to run |
Conclusion
That’s the stack: Proxmox, TrueNAS, Authentik, NetBird, GitLab. Free, boring in the best way, and it’s now the backbone that NUCC, UCI, CSULA, CSUN, CSUF, and WRCCDC all run on, and that NUCC has helped get real hardware behind so more schools can stand up a production grade environment for teaching people how to defend a network.
None of this exists without the people actually building this software for free, on their own time, and I don’t think that gets said enough. I’ve gotten to meet some of the Authentik folks at DEF CON, and every time it’s the same feeling: these are people who genuinely give a fuck about the thing they built, answering stupid ass questions from an idiot at 11pm on a Discord server, and spending so much time and effort to keep something available so that we can use it to do cool shit. The more proprietary software I have to deal with at work that barely does the one thing it’s licensed to do, the more I appreciate how much better this stuff is, built by people who aren’t trying to squeeze a renewal fee out of you. This entire club has been able to teach hundreds of students real infrastructure and security skills because a handful of maintainers decided to give their work away.