最近在阿里轻量云买了一些机器, 同样都是1C/0.5G, Ubuntu 20.04 的机器运行非常正常, Ubuntu 22.04 的机器却隔一段时间就没有响应了. 具体表现为能ping通, 但是ssh登录会超时失败, 宿主机监控CPU/磁盘IO大涨. 登录VNC能看到类似这样的提示:
1 2 3 4 5 6 7 8 9
| [44211.553196] Out of memory: Killed process 95466(apt-check) ... shmem-rss:0KB, UID:0 pgtables:300KB dom_score_adj:0 [44213.738115] systemd[1]: Failed to start Refresh fwupd metadata and update motd. [63358.618014] Out of memory: Killed process 123118(apt-check) total-vm:190376KB ... UID:0 pgtables:364kB oom_score_adj:0 [63359.212458] systemd[1] : Failed to start Dailyapt download activities. [74055.581349] Out of memory: Killed process 126756 (apt-check) total-vm:190376kB, ... UID:0 pgtables:376kB oom_score_adj:0 [121996.542525] Out of memory: Killed process 210249 (apt-check) total-vm:100788kB, ... UID:0 pgtables:228kB oom_score_adj:0 [121996.882131] systemd[1]: snapd.service: Watchdogtimeout (limit 5min) [121997.311208] systemd[1]: Failed to start Dailyapt download activities. [179235.303036] Out of memory: Killed process 292938(apt-check) total-vm:190376KB, ... UID:0 pgtables:372kB oom_score_adj:0
|
而这台机器已经关闭了apt自动更新, 一番搜索之后发现snapd可能会引起这个问题:
snap list
1 2 3 4
| Name Version Rev Tracking Publisher Notes core20 20240416 2318 latest/stable canonical✓ base lxd 5.0.3-80aeff7 29351 5.0/stable/… canonical✓ - snapd 2.63 21759 latest/stable canonical✓ snapd
|
- 删除snap各个包(有依赖关系)
1 2 3
| sudo snap remove --purge lxd sudo snap remove --purge core20 sudo snap remove --purge snapd
|
- 删除snap
1 2
| sudo apt remove snapd sudo apt purge snapd
|
- 添加apt配置文件防止snapd重新被安装回来
sudo vim /etc/apt/preferences.d/nosnap.pref
1 2 3
| Package: snapd Pin: release a=* Pin-Priority: -10
|
- 清除apt缓存, 重新更新一下(可选)
sudo apt clean && sudo apt update
在完全删除掉snapd之后, 目前机器已经正常运行了两三天…
后续
后来发现还是不太行, 解决方案是给这个内存超级小的机器加上Swap. 因为阿里云没创建swap, 而且还把swappiness设置成了0! 为防止奇怪的事情发生, 弄成crontab脚本每分钟跑一下好了.
创建并启用swap
1 2 3 4
| dd if=/dev/zero of=/swap.img bs=1M count=1024 chmod 600 /swap.img mkswap /swap.img swapon /swap.img
|
添加分钟级任务 sudo crontab -e
1 2 3 4
| @reboot swapon -s | grep -q swap || swapon /swap.img @reboot echo 60 > /proc/sys/vm/swappiness * * * * * swapon -s | grep -q swap || swapon /swap.img * * * * * echo 60 > /proc/sys/vm/swappiness
|
参考
Terminate unattended-upgrades or whatever is using apt in ubuntu 18.04 or later editions
How to Remove Snap Packages in Ubuntu Linux
How do I configure swappiness?
How to read oom-killer syslog messages?
How can I check if swap is active from the command line?
Linux Partition HOWTO: 9. Setting Up Swap Space
How to Clear RAM Memory Cache, Buffer and Swap Space on Linux
Swappiness: What it Is, How it Works & How to Adjust