Set up systemd-oomd and limit gitlab memory usage
Current memory usage of gitlab services:
gitlab: 3.8G (12%)
gitlab-pages: 19.7M (~0%)
gitlab-workhorse: 46.4M (~0%)
gitlab-runner: 58.9M (~0%)
gitlab-sidekiq: 1.0G (~3%)
total: (16%)
We might want to test that memory usage doesn't go over 50% during a CI job before deploying this.
I haven't tested this config at all, although I did test a similar system in a VM and it had reasonable behavior:
systemd.services = let
names = [ "hog-1" "hog-2" ];
in builtins.listToAttrs (builtins.map (name: {
name = name;
value = {
serviceConfig = {
ExecStart = "${pkgs.stress-ng}/bin/stress-ng --vm-bytes 256M -m 1";
Restart = "on-failure";
Slice = "system-hogs.slice";
};
};
}) names);
systemd.slices."system-hogs" = {
sliceConfig = {
MemoryHigh = "300M";
MemoryMax = "500M";
};
};
systemd.oomd.enable = true;
systemd.slices."system" = {
sliceConfig = {
ManagedOOMMemoryPressure = "kill";
ManagedOOMMemoryPressureLimit = "60%";
};
};
systemd.slices."user" = {
sliceConfig = {
ManagedOOMMemoryPressure = "kill";
ManagedOOMMemoryPressureLimit = "40%";
};
};
systemd.slices."root" = {
sliceConfig = {
MemoryHigh = "95%";
};
};