怀念老歌专场了属于是

罗百吉 - I Miss You

据说这首歌能勾起很多人回忆?-《哎呀》

这才是《不是因为寂寞才想你》的原唱,很多80、90后的青春回忆!

汤潮 - 狼爱上羊

【怀旧经典】庞龙《两只蝴蝶》超级王炸神曲!火遍大江南北的神曲!

【4K 无损+伴奏】马天宇 - 《该死的温柔》

【怀旧经典】王强《秋天不回来》超级王炸神曲!火遍大江南北的神曲!

【怀旧经典】誓言《求佛》超级王炸神曲!火遍大江南北的神曲!

【怀旧经典】徐誉滕《等一分钟》王炸神曲!铃声真的动了!

这才是《老鼠爱大米》的原版,听了好像回到童年。

【4K 童年神曲】郭美美 - 不怕不怕 MV 修复版

【回忆杀】-胡杨林《香水有毒》,经典永流传!

【怀旧经典】刀郎《2002年的第一场雪》王炸神曲!2004年华语乐坛系列!

Imagine Dragons - Bones (Official Lyric Video) 阿姨压一压

崔子格 - 卜卦「 祈求別再變卦 」♪Karendaidai♪

Neil Cicierega - Daft Mouth

Bật Tình Yêu Lên Remix (Deep House)

#古風 殘雪、林玉涵-苦海邊◖夢回千年,最後你成了我一生心魔◗ [Official Lyric Video] 官方動態歌詞版

Tiësto vs Diplo - C’Mon

[for Orzmic] MYUKKE. - BiBiC HEART

FlyBoy & The Onyx Twins - I Will Be Ok (ft. Coby Grant)

Mae Stephens - If We Ever Broke Up (Official Video)

「Cyberangel」——《崩坏3》印象曲(演唱者:Hanser)

往期优秀作品推荐

2023年5-6月

经过接近7个月的测试, 目前两级存储中共有5.1亿条数据, 总数据量约230GB. SSD中热数据量约为10%

背景: 因为SSD存储太贵, 而且一般只有最近14天的数据需要经常查询, 因此把14天外的数据送到HDD里保存节约成本.

  1. 需要新增存储配置文件 storage.xml 其中 cold_disk 为HDD存储配置, cold_volume为HDD存储策略
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<clickhouse>
<storage_configuration>
<disks>
<default>
</default>
<cold_disk>
<path>/var/lib/clickhouse-hdd/</path>
</cold_disk>
</disks>
<policies>
<default>
<volumes>
<default>
<disk>default</disk>
</default>
<cold_volume>
<disk>cold_disk</disk>
</cold_volume>
</volumes>
</default>
</policies>
</storage_configuration>
</clickhouse>
  1. 给数据表添加TTL策略, 默认去default, 15天以上的去cold_volume
1
alter table database_name.table_name modify TTL create_time TO VOLUME 'default', create_time + INTERVAL 15 DAY TO VOLUME 'cold_volume'
  1. 刷新TTL策略, 强制冷数据移动
1
alter table database_name.table_name materialize ttl

参考资料

ClickHouse 单机docker多盘搭建教程

Amplifying ClickHouse Capacity with Multi-Volume Storage (Part 1)

Amplifying ClickHouse Capacity with Multi-Volume Storage (Part 2)

Putting Things Where They Belong Using New TTL Moves 中文翻译,但是水平一般: 【ClickHouse 技术系列】- 使用新的 TTL move,将数据存储在合适的地方

Manage Data with TTL (Time-to-live)

TrueNAS SCALE 22.04.3 运行一直很稳定, 有一天开始突然报错CRITIAL:

1
Error: (401) Reason: Unauthorized HTTP response headers: < CIMultiDictProxy('Audit-Id': '7b4e8f99-ae3e-415e-b80b-d5ab72dadd85', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Sun, 28 May 2023 07:27:45 GMT', 'Content-Length': '129') > HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

当时没太当回事, 以为是官方仓库挂了. 一个多月后再检查还是这样, 进shell看了一下journal发现是x509证书过期了:

1
May 29 03:23:29 truenas k3s[11305]: E0529 03:23:29.178174   11305 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2023-05-29T03:23:29+08:00 is after 2023-04-16T16:35:58Z, verifying certificate SN=..., SKID=, AKID=... failed: x509: certificate has expired or is not yet valid: current time 2023-05-29T03:23:29+08:00 is after 2023-04-16T16:35:58Z]"

shell直接运行k3s kubectl get node也会报错提示 Unauthorized.

解决方法很简单, 重启k3s和docker服务就行 (当然重启主机也可以, 只是会中断当前的读写):

1
2
systemctl stop k3s docker
systemctl start docker k3s

参考

Truecharts: Docker pull error x509: certificate valid for…

沉迷越南土嗨

Đau Ở Đây Này (Orinn Remix) - NAL | Ta Như Làn Mây Trôi Về Cuối Bầu Trời Remix TikTok Hay Nhất 2022

Khuất Lối (Orinn Remix) - H Kray | Anh Đi Về Bóng Tối Khuất Lối | Nhạc Trẻ Remix Hot Tiktok 2022

Anh Đã Lạc Vào Remix VER 2 - (Green, Đại Mèo Remix)anh đã lạc vào cả một bầu trời yêu dấu này remix

BIẾT ÔNG THƯƠNG KHÔNG? Thương cho Tấm Thân Cơ Hàn HOT TIK TOK, Tấm Lòng Son Remix -H-Kray x Đại Mèo

Thôi Quên Đi (Orinn Remix) - TVk x PHÁT HUY T4 | Nhạc Trẻ Remix Căng Cực Gây Nghiện Hay Nhất 2022

Chỉ Bằng Cái Gật Đầu Remix - (Yan Nguyễn x Đại Mèo) Có lẽ cần một cái gật đầu xin cha mẹ về bên anh

Tấm Lòng Son Remix - H-Kray x Đại Mèo Remix I giọt buồn vương trên màu mắt ai remix hot TIKTOK

Hoa Cưới (H2O Remix) - Đạt Long Vinh | Rồi Người Rời Bước Thật Mau Mặc Vào Tà Áo Nàng Dâu Remix

Vương Vấn Remix - Qinn Remix x TVk x Hana Cẩm Tiên - Chắc Chỉ Mỗi Anh Vẫn Còn Thương Remix TikTok

Nặng Tình Hay Nhẹ Lòng - WRC REMIX - Tống Gia Vỹ

Trót Trao Duyên (H2O Remix) - NB3 Hoài Bảo | Và Anh Biết Duyên Mình Đã Lỡ Hot TikTok Remix

CÔ BỎ ĐI HẾT BAO NHIÊU MỘNG MƠ - Phận Tàn REMIX - TVk x Phát Lee (Đại Mèo Remix)

Thuyền Quyên (AIR Remix) - Diệu Kiên ♫ Dặm Ngàn Thiên Lý Tiễn Người Đi Remix | Hot Trend TikTok

Xin Má Rước Dâu (AIR Remix) - Diệu Kiên ♫ Anh Ơi Nắng Mưa Dãi Dầu Về Nhà Xin Má Rước Con Dâu Remix

♬ PHẬN DUYÊN LỠ LÀNG - PHÁT HUY T4 X TRUZG ( KAINE X HHD REMIX ) | @hhdmusicofficial

♬ CHẠNH LÒNG THƯƠNG CÔ 2 - Huy Vạc ( WinT Remix ) | Nhớ Đeo Tai Nghe @hhdmusicofficial

THÔI EM ĐỪNG NẶNG LÒNG ANH TA REMIX VINAHOUSE / EM NÊN DỪNG LẠI - KHANG VIỆT X KONIS WILLIAMS

Đào Nương (Ciray Remix) - Hoàng Vương | Gặp lại cô ta còn đau đáu rỏ thêm máu lên trang thơ nhàu…

Lý Do Là Gì (AIR Remix) - Nguyễn Vĩ ♫ Nước Mắt Anh Tuôn Theo Màn Đêm Buông Xuống Remix TikTok

Da Da Da (Remix by Mikis)

黄梅戏 Remix 慕容晓晓 DJ版 | 抖音 || 我的公子又在何方 《女驸马》 为救李郎离家园

往期优秀作品推荐

2023年3-4月

卸载主机安全、监控等组件

1
2
3
/usr/local/qcloud/stargate/admin/uninstall.sh
/usr/local/qcloud/YunJing/uninst.sh
/usr/local/qcloud/monitor/barad/admin/uninstall.sh

卸载自动化助手, 参考 下载之后运行uninstall.sh

卸载新的一键登录等组件 (参考用, 主要需要识别出哪些是腾讯的组件)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#!/bin/bash 
#fuck tx process
rm -rf /usr/local/sa
rm -rf /usr/local/agenttools
rm -rf /usr/local/qcloud
process=(sap100 secu-tcs-agent sgagent64 barad_agent agent agentPlugInD pvdriver )
for i in ${process[@]}
do
for A in $(ps aux |grep $i |grep -v grep |awk '{print $2}')
do
kill -9 $A
done
done
chkconfig --level 35 postfix off
service postfix stop
echo ''>/var/spool/cron/root
echo '#!/bin/bash' >/etc/rc.local

参考

完整优雅的卸载腾讯云云服务器安全监控组件云镜

情歌对唱《敖包相恋》你爱我嘛, 当然爱啦

《Yesterday Once More》《昨日重現》英文歌中文譯

Lemaitre - Closer ft. Jennie A. (UK Version)

Lemaitre - Smoke (Lyric Video)

【imase】NIGHT DANCER(MV)

TONES AND I - DANCE MONKEY (OFFICIAL VIDEO)

Klaypex ft. GRÉTA - Robot Love

Brothers - The Moon

Funky Russian Train/Maxwells song for 2 minutes and 40 seconds

DuckTales Music (NES) - The Moon Theme

难得真兄弟 (DJ小鱼儿版)

往期优秀作品推荐

2023年1-2月

需求: 对某个network下的全部容器执行特殊流量策略, 例如调整默认gateway

一种朴素的方式是在容器启动后, 服务启动前, 通过在host侧使用ip netns或在容器中使用ip route命令调整默认gateway. 但这些方法都比较重, 对容器创建过程有很高的要求, 而且需要保证容器内服务能够配合改造.

我们决定采用另一种方式, 首先通过sudo podman network create mustredirect创建一个network. 可以看到提示CNI配置文件在这里 /etc/cni/net.d/mustredirect.conflist

配置文件如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
{
"cniVersion": "0.4.0",
"name": "mustredirect",
"plugins": [
{
"type": "bridge",
"bridge": "cni-podman5",
"isGateway": true,
"ipMasq": true,
"hairpinMode": true,
"ipam": {
"type": "host-local",
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"ranges": [
[
{
"subnet": "10.89.4.0/24",
"gateway": "10.89.4.1"
}
]
]
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
},
{
"type": "firewall",
"backend": ""
},
{
"type": "tuning"
},
{
"type": "dnsname",
"domainName": "dns.podman",
"capabilities": {
"aliases": true
}
}
]
}

我们需要编辑这个文件, 调整 ipMasqfalse, 这样能够阻止CNI生成iptables MASQUERADE规则, 从而保证我们的流量在打到网关的时候, 源IP为容器的IP而不是节点的IP, 这样可以方便我们在网关层做流量拆分.

之所以需要手动编辑是因为podman v3.4.4目前还不支持在创建的时候指定ipMasq选项, 直接用--opt指定会报错.

接下来创建iptables规则, 因为CNI组件默认调整的是filter和nat表, 为了保证我们的流量策略优先级更高, 我们会把如下流量策略加入到raw表:

sudo iptables -t raw -A PREROUTING -s 10.89.4.0/24 -j MARK --set-mark 0x888

然后添加ip rule规则:

sudo ip rule add fwmark 0x888 lookup 0x888

然后为table 0x888添加默认网关

sudo ip route add default via <独立网关IP> table 0x888

在网关服务器上, 先给10.89.4.0/24的流量添加路由:

sudo ip route add 10.89.4.0/24 via <运行容器的NodeIP>

接下来添加流量策略:

sudo ip rule add 10.89.4.0./24 lookup table 0x888

为table 0x888添加默认网关, 此处假设要求流量必须从gre0发送给上游

sudo ip route add default gre0 ens18 table 0x888

最后, 在网关层添加iptables规则, 保证上游收到的IP不再是容器的IP. 当然这里如果是和上游一起组网, 也可以考虑不添加MASQUERADE, 而是直接用OSPF协议同步路由

sudo iptables -t nat -A POSTROUTING -o gre0 -j MASQUERADE

这样, 我们就能保证来自mustredirect网络下的容器的流量, 全部经过这个新的网关发往上游gre0对端, 且不需要单独配置回程. 所有挂在mustredirect网络下的容器不需要再单独设置即可满足需求. 而且这里完全不再需要再有一层类似ipip/gre/vxlan的封装.

注: 编辑CNI网络文件时请保证没有容器连接到这个网络, 即Podman不会调用CNI创建这个interface.

这个配置流程的流程受到了 K8S Cilium BGP 路由的启发.

参考

How Container Networking Works: Practical Explanation

CNI - bridge plugin

podman - cni/README.md

podman - cmd/podman/networks/create.go 可以看到目前还无法支持ipMasq选项

JETFIRE & Karmatek - Living On The Edge (Official Audio)

R3HAB & Skytech - Everything

Jay Hardway - Electric Elephants (Official Music Video)

7UBO - Furia

Akiko Wada - YONA YONA DANCE

Nightcore - My Forever

Alex Aster - Divine (Lyrics)

Light Dance - sakanaction

サカナクション / 多分、風。 -Music Video-

Ummet Ozcan - Xanadu (Mongolian Techno)

M83 - Midnight City (Lyrics)

La La La (VK Remix)

Legends Never Die: Remix (ft. Alan Walker) | Worlds 2017 - League of Legends

海鸟飞鱼 - 思念绕指尖 (DJ名龙版)

往期优秀作品推荐

2022年11-12月

内网dns服务器可以参考的一些配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
# Never forward plain names (without a dot or domain part)
# 不包含点(.)的域名不要发给上游DNS服务器(不会流出当前节点)
domain-needed

# Never forward addresses in the non-routed address spaces.
# 无法路由的地址不要发给上游DNS服务器(不会流出当前节点)
bogus-priv

# If you don't want dnsmasq to read /etc/resolv.conf or any other
# file, getting its servers from this file instead (see below), then
# uncomment this.
# 不读取/etc/resolv.conf, 因为里面只有namserver 127.0.0.1
no-resolv

# Add other name servers here, with domain specs if they are for
# non-public domains.
# 对于解析不了的域名, 转发某个上游DNS服务器
server=192.168.50.1

# Add local-only domains here, queries in these domains are answered
# from /etc/hosts or DHCP only.
# 对于.in39结尾的域名都视为内网, 不转发给上游DNS服务器.
local=/in39/

# Add domains which you want to force to an IP address here.
# The example below send any host in double-click.net to a local
# web-server.
# 解析, 泛解析域名到IP, 如下*.helloworld.in39和helloworld.in39都会解析到这个IP
# 域名本身也可以加通配符来做字符串匹配解析.
address=/helloworld.in39/192.168.50.1

# If you don't want dnsmasq to read /etc/hosts, uncomment the
# following line.
# 不要读取/etc/hosts
no-hosts

# or if you want it to read another file, as well as /etc/hosts, use
# this.
# 但是读取以下指定的文件作为hosts内容 格式和hosts保持一致. 多个IP可以解析到同一个域名, dnsmasq可以正确解析.
addn-hosts=...

# Include another lot of configuration options.
# conf-file=/etc/dnsmasq.more.conf
# 可以把配置文件分离到其他文件或文件夹里, 可以有多个
conf-dir=...

参考

dnsmasq - ArchWiki

Wildcard subdomains with dnsmasq

Stop DNSMasq From Forwarding Local Hostnames

Assign multiple IPs to 1 Entry in hosts file

Is there a way to use a specific DNS for a specific domain?

有个简短的视频可以参考 Configuring DNS With Dnsmasq and Ubuntu Server

问题现象

需求是将服务的UDP流量从机器A切换到机器B. 路由器操作前机器A和机器B相关服务已准备就绪. 切换期间上游会一直有流量过来.

路由器设置端口转发, UDP协议, 外部端口保持不变的情况下改变内部IP, 保存后不生效, UDP包仍然会发送到原来的内网IP.

在机器A上运行tcpdump:

1
2
06:13:05.130930 IP (tos 0x28, ttl 61, id 60828, offset 0, flags [none], proto UDP (17), length 176)
##.##.##.##.51820 > ##.##.##.##.51820: UDP, length 148

注意此时机器A上已没有服务监听在目标端口, 已通过iptables DROP来源包, 否则会有ICMP不可达报文

1
2
3
4
05:48:51.052956 IP (tos 0xc8, ttl 64, id 24675, offset 0, flags [none], proto ICMP (1), length 204)
##.##.##.## > ##.##.##.##: ICMP ##.##.##.## udp port 51820 unreachable, length 184
IP (tos 0x28, ttl 61, id 15781, offset 0, flags [none], proto UDP (17), length 176)
##.##.##.##.51820 > ##.##.##.##.51820: UDP, length 148

在机器B(新的目标机)运行tcpdump接收不到包.

问题排查

机器A和B本地排查无果, 登录路由器进行排查.

运行conntrack -L:

1
2
udp      17 47 src=[远端机器的IP] dst=[路由器公网侧IP] sport=51820 dport=53820 src=[机器A的IP] dst=[远端机器的IP] sport=51820 dport=51820 [ASSURED] mark=0 use=1
conntrack v1.4.5 (conntrack-tools): 686 flow entries have been shown.

运行iptables -t nat -vnL

1
0    0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53820 to:192.168.50.3:53820

可以看到端口转发配置是生效的, 但是因为有conntrack规则的存在所以后续来的包并没有被当成”新的链接”走iptables, 而是继续按照conntrack中的规则进行转发.

这里比较怀疑这个特定版本的conntrack可能经过魔改, 查到的标准是180秒内如果没有回包, conntrack规则就应该被移除. 但实际上在路由器上可以看到:

1
2
3
4
5
6
udp      17 1 src=##.##.##.## dst=##.##.##.## sport=51820 dport=53820 src=##.##.##.## dst=##.##.##.## sport=51820 dport=51820 [ASSURED] mark=0 use=1
conntrack v1.4.5 (conntrack-tools): 710 flow entries have been shown.
udp 17 0 src=##.##.##.## dst=##.##.##.## sport=51820 dport=53820 src=##.##.##.## dst=##.##.##.## sport=51820 dport=51820 [ASSURED] mark=0 use=1
conntrack v1.4.5 (conntrack-tools): 710 flow entries have been shown.
udp 17 179 src=##.##.##.## dst=##.##.##.## sport=51820 dport=53820 src=##.##.##.## dst=##.##.##.## sport=51820 dport=51820 [ASSURED] mark=0 use=1
conntrack v1.4.5 (conntrack-tools): 707 flow entries have been shown.

很明显, 规则定时归0时没有被删除而是重置了倒计时.

解决方案

在路由器上, 使用命令删除这条规则: conntrack -D -p udp --dport 53820

删除后马上在机器B的tcpdump上就可以看到来自远端机器的UDP流量了.

参考

portmap: delete UDP conntrack entries on teardown · Issue #123 · containernetworking/plugins

在CNI github issue里找到了一个类似的问题, 带udp端口转发的pod移除的时候需要手动调conntrack删除掉NAT规则, 否则流量将无法分配到新的pod上. 但这种场景里路由器(上层NAT设备)一般是不在控制范围内的, 可能除了在远端发起换端口之外没有任何办法了…

netfilter: Kill unreplied conntracks by ICMP errors

这里有一个patch提议说可以用ICMP错误回包来剔除掉netfilter conntrack里无效的规则, 但应该没有被merge到linux kernel里.

Linux Packet Filtering and iptables - 7.5. UDP connections

Iptables Tutorial 1.2.1 - 7.5. UDP connections

The conntrack-tools user manual

Conntrack tales - one thousand and one flows

Connection Tracking (conntrack): Design and Implementation Inside Linux Kernel