在GCP上配置IPSec隧道

  1. 创建对等VPN网关 (Peering VPN Gateway)

  1. 创建Cloud VPN网关和隧道, 此时就是客户端的角色,这里要选择网卡数量,最多两个网卡,每个网卡分配一个公网IP

此时还不能配置BGP对话, 需要先建立起连接才可以

搭建IPSec链接

安装 Strongswan (IPSec VPN)

1
sudo apt install strongswan strongswan-pki

配置ipsec, 配置文件路径 /etc/ipsec.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
config setup
charondebug="all"
uniqueids=yes
strictcrlpolicy=no

conn %default
ikelifetime=600m # 36,000 s
keylife=180m # 10,800 s
rekeymargin=3m
keyingtries=3
keyexchange=ikev2
mobike=no
ike=aes256gcm16-sha512-modp4096 # 这里的参数需要跟GCP平台匹配
esp=aes256gcm16-sha512-modp8192 # 这里的参数需要跟GCP平台匹配
authby=psk

conn net-net1
leftupdown="/var/lib/strongswan/ipsec-vti.sh 0 169.254.232.77/32 169.254.232.78/32" # 这里是p2p通道的GCP侧IP和本端IP
left=10.0.8.4 # In case of NAT set to internal IP, e.x. 10.164.0.6
leftid=10.0.8.4
leftsubnet=0.0.0.0/0
leftauth=psk
right={GCP平台的公网IP}
rightid=%any
rightsubnet=0.0.0.0/0
rightauth=psk
type=tunnel
# auto=add - means strongSwan won't try to initiate it
# auto=start - means strongSwan will try to establish connection as well
# Note that Google Cloud will also try to initiate the connection
auto=start
# dpdaction=restart - means strongSwan will try to reconnect if Dead Peer Detection spots
# a problem. Change to 'clear' if needed
dpdaction=restart
mark=%unique

conn net-net2
leftupdown="/var/lib/strongswan/ipsec-vti.sh 1 169.254.155.53/32 169.254.155.54/32" # 同上
left=10.0.8.4 # In case of NAT set to internal IP, e.x. 10.164.0.6
leftid=10.0.8.4
leftsubnet=0.0.0.0/0
leftauth=psk
right={GCP平台的公网IP}
rightid=%any
rightsubnet=0.0.0.0/0
rightauth=psk
type=tunnel
# auto=add - means strongSwan won't try to initiate it
# auto=start - means strongSwan will try to establish connection as well
# Note that Google Cloud will also try to initiate the connection
auto=start
# dpdaction=restart - means strongSwan will try to reconnect if Dead Peer Detection spots
# a problem. Change to 'clear' if needed
dpdaction=restart
mark=%unique

其中vti脚本内容如下, 路径 /var/lib/strongswan/ipsec-vti.sh (源自网络,见参考)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
#!/bin/bash
set -o nounset
set -o errexit

IP=$(which ip)

PLUTO_MARK_OUT_ARR=(${PLUTO_MARK_OUT//// })
PLUTO_MARK_IN_ARR=(${PLUTO_MARK_IN//// })

VTI_TUNNEL_ID=${1}
VTI_REMOTE=${2}
VTI_LOCAL=${3}

LOCAL_IF="${PLUTO_INTERFACE}"
VTI_IF="vti${VTI_TUNNEL_ID}"
# GCP's MTU is 1460, so it's hardcoded
GCP_MTU="1460"
# ipsec overhead is 73 bytes, we need to compute new mtu.
VTI_MTU=$((GCP_MTU-73))

case "${PLUTO_VERB}" in
up-client)
${IP} link add ${VTI_IF} type vti local ${PLUTO_ME} remote ${PLUTO_PEER} okey ${PLUTO_MARK_OUT_ARR[0]} ikey ${PLUTO_MARK_IN_ARR[0]}
${IP} addr add ${VTI_LOCAL} remote ${VTI_REMOTE} dev "${VTI_IF}"
${IP} link set ${VTI_IF} up mtu ${VTI_MTU}

# Disable IPSEC Policy
sysctl -w net.ipv4.conf.${VTI_IF}.disable_policy=1

# Enable loosy source validation, if possible. Otherwise disable validation.
sysctl -w net.ipv4.conf.${VTI_IF}.rp_filter=2 || sysctl -w net.ipv4.conf.${VTI_IF}.rp_filter=0

# If you would like to use VTI for policy-based you should take care of routing by yourselv, e.x.
#if [[ "${PLUTO_PEER_CLIENT}" != "0.0.0.0/0" ]]; then
# ${IP} r add "${PLUTO_PEER_CLIENT}" dev "${VTI_IF}"
#fi
;;
down-client)
${IP} tunnel del "${VTI_IF}"
;;
esac

# Enable IPv4 forwarding
sysctl -w net.ipv4.ip_forward=1

# Disable IPSEC Encryption on local net
sysctl -w net.ipv4.conf.${LOCAL_IF}.disable_xfrm=1
sysctl -w net.ipv4.conf.${LOCAL_IF}.disable_policy=1

调整 charon 配置, 不要安装路由 (下一步由bird2配置具体的路由, 否则会像WireGuard一样安装一个0.0.0.0/0的路由) /etc/strongswan.d/vti.conf 这里其实有点像wg的Table=off

1
2
3
4
charon {
# We will handle routes by ourselves
install_routes = no
}

配置防火墙/端口转发(如有需要), ipsec使用: 500/udp,4500/udp,4510/udp,4511/udp

通过命令启动/关闭/查看ipsec隧道

启动: sudo ipsec start

关闭: sudo ipsec stop

查看状态 sudo ipsec statusall (也可以使用 sudo ipsec status)

看到ESTABLISHED字样就表明已经建立连接了 (这里只能看到一个因为写文章的时候已经开始回收测试环境了)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Status of IKE charon daemon (strongSwan 5.9.5, Linux 5.15.0-107-generic, x86_64):
uptime: 13 minutes, since Jul 25 14:04:56 2024
malloc: sbrk 3100672, mmap 0, used 1427696, free 1672976
worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 4
loaded plugins: charon aesni aes rc2 sha2 sha1 md5 mgf1 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf gmp agent xcbc hmac gcm drbg attr kernel-netlink resolve socket-default connmark stroke updown eap-mschapv2 xauth-generic counters
Listening IP addresses:
10.0.8.4
169.254.232.78
Connections:
net-net1: 10.0.8.4...34.128.44.254 IKEv2, dpddelay=30s
net-net1: local: [10.0.8.4] uses pre-shared key authentication
net-net1: remote: uses pre-shared key authentication
net-net1: child: 0.0.0.0/0 === 0.0.0.0/0 TUNNEL, dpdaction=restart
net-net2: 10.0.8.4...<reducted> IKEv2, dpddelay=30s
net-net2: local: [10.0.8.4] uses pre-shared key authentication
net-net2: remote: uses pre-shared key authentication
net-net2: child: 0.0.0.0/0 === 0.0.0.0/0 TUNNEL, dpdaction=restart
Security Associations (1 up, 0 connecting):
net-net1[4]: ESTABLISHED 13 minutes ago, 10.0.8.4[10.0.8.4]...<reducted>
net-net1[4]: IKEv2 SPIs: <reducted>, pre-shared key reauthentication in 9 hours
net-net1[4]: IKE proposal: AES_GCM_16_256/PRF_HMAC_SHA2_512/MODP_4096
net-net1{1}: INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: <reducted>
net-net1{1}: AES_GCM_16_256, 5986 bytes_i, 6007 bytes_o (96 pkts, 4s ago), rekeying in 2 hours
net-net1{1}: 0.0.0.0/0 === 0.0.0.0/0

在GCP上配置BGP链路

创建高可用VPN, 本质是一个隧道组,要实现HA,最少需要 2 中的两个网卡每个网卡配置一条隧道到1,最多可以配置 n*m 个隧道

搭建BGP链路

安装BIRD2 sudo apt install bird2

编写bird2配置 /etc/bird/bird.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
log syslog all;
debug protocols all;

protocol device {
scan time 10;
}

protocol direct {
ipv4; # Connect to default IPv4 table
}

protocol kernel {
ipv4 { # Connect protocol to IPv4 table by channel
import none;
export all; # Export to protocol. default is export none
};
}

protocol static {
ipv4;
route 192.168.48.0/24 via 10.181.0.2;
route 169.254.232.77/32 via 192.254.232.78;
}

protocol bgp gcp_vpc_a_tun1 {
local 169.254.232.78 as 65003; # 本端ASN
neighbor 169.254.232.77 as 64520; # 对端ASN
multihop;
keepalive time 20;
hold time 60;
graceful restart aware;
ipv4 {
import filter {
gw = 169.254.232.77; # 不知道为什么一定要加这个
accept;
};
import limit 10 action warn;
export filter{
if (net ~ 192.168.0.0/16) then accept; # 这里是输出路由给GCP时的过滤
else reject;
};
export limit 10 action warn;
};
}

protocol bgp gcp_vpc_a_tun2 {
local 169.254.155.54 as 65003; # 本端ASN
neighbor 169.254.155.53 as 64520; # 对端ASN
multihop;
keepalive time 20;
hold time 60;
graceful restart aware;
ipv4 {
import filter {
gw = 169.254.155.53;
accept;
};
import limit 10 action warn;
export filter{
if (net ~ 192.168.0.0/16) then accept;
else reject;
};
export limit 10 action warn;
};
}

重加载bird配置 sudo birdc configure

查看bird协议状态 sudo birdc show protocol all

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
BIRD 2.0.8 ready.

<中间省略>

gcp_vpc_a_tun1 BGP --- up 14:04:57.581 Established
BGP state: Established
Neighbor address: 169.254.232.77
Neighbor AS: 64520
Local AS: 65003
Neighbor ID: 169.254.232.77
Local capabilities
Multiprotocol
AF announced: ipv4
Route refresh
Graceful restart
4-octet AS numbers
Enhanced refresh
Long-lived graceful restart
Neighbor capabilities
Multiprotocol
AF announced: ipv4
Route refresh
Graceful restart
Restart time: 60
Restart recovery
AF supported: ipv4
AF preserved: ipv4
4-octet AS numbers
Session: external multihop AS4
Source address: 169.254.232.78
Hold timer: 43.240/60
Keepalive timer: 1.847/20
Channel ipv4
State: UP
Table: master4
Preference: 100
Input filter: (unnamed)
Output filter: (unnamed)
Import limit: 10
Action: warn
Export limit: 10
Action: warn
Routes: 1 imported, 0 exported, 1 preferred
Route change stats: received rejected filtered ignored accepted
Import updates: 1 0 0 0 1
Import withdraws: 0 0 --- 0 0
Export updates: 3 1 2 --- 0
Export withdraws: 0 --- --- --- 0
BGP Next hop: 169.254.232.78
IGP IPv4 table: master4

gcp_vpc_a_tun2 BGP --- start 14:04:56.161 Active Socket: Connection closed 同上,如果有两个对端且配置正确这里会是Established,但是对端已经被回收了
BGP state: Active
Neighbor address: 169.254.155.53
Neighbor AS: 64520
Local AS: 65003
Connect delay: 2.781/5
Last error: Socket: Connection closed
Channel ipv4
State: DOWN
Table: master4
Preference: 100
Input filter: (unnamed)
Output filter: (unnamed)
Import limit: 10
Action: warn
Export limit: 10
Action: warn
IGP IPv4 table: master4

localnet OSPF master4 up 14:04:56.161 Alone
Channel ipv4
State: UP
Table: master4
Preference: 150
Input filter: (unnamed)
Output filter: (unnamed)
Routes: 0 imported, 3 exported, 0 preferred
Route change stats: received rejected filtered ignored accepted
Import updates: 0 0 0 0 0
Import withdraws: 0 0 --- 0 0
Export updates: 4 0 0 --- 4
Export withdraws: 0 --- --- --- 0

没问题的话就能看到GCP推送过来的路由了 ip route

1
2
3
4
5
6
...
10.0.1.0/24 via 169.254.232.77 dev vti0 proto bird metric 32
...
169.254.232.77 dev vti0 proto kernel scope link src 169.254.232.78
169.254.232.77 dev vti0 proto bird scope link metric 32
...

参考

How to set up a VPN between strongSwan and Cloud VPN

Using Strongswan to setup site to site IPsec VPN between GCP and Digital Ocean

Configuring Site-to-Site IPSec VPN on Ubuntu using Strongswan

Figuring out how ipsec transforms work in Linux

Establish VPN tunnel for in-house machine to access GCP network

KB: Connecting OpenWRT/LEDE router to Azure Virtual Network Gateway (IKEv2)

Secure site-to-site connection with Linux IPsec VPN

Google Cloud HA VPN interoperability guide for AWS

How does IPsec VPN really work?

BGP 对等会话

Google史一样的文档: Create two fully configured HA VPN gateways that connect to each other | Establish BGP sessions

GCP Networking: Part 2 Cloud Router

Foo over UDP

IPsec vs. WireGuard

The Noise Protocol Framework

rp_filter - Sysctl Explorer

howto/Bird2 - dn42 虽然是dn42的教程但是非常管用

Gym Class Heroes - Make me your radio (Stereo Hearts) (Lyrics)

She Wolf Youtube上一大堆原神MMD的BGM

Sea of Tranquility - Bemax F1的小曲 《F1の小曲》《杆位の小曲》《最速の小曲》Sea of Tranquility-Bemax 应用: (无水印)F1赛车手出场meme报幕遮挡原版素材

Chinese 2024最火歌曲DJ ✔2024最火歌曲DJ Remix 抖音版 🎶 最好的音樂Chinese DJ remix 👍 Douyin Dj抖音版2024

不知道为啥缺氧没做这个功能, 创意工坊里也没有做这个的人, 只好自己写个脚本来做了. 特别感谢 @RoboPhred 提供的 oni-save-parser

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
const { readFileSync, writeFileSync } = require("fs");
const { parseSaveGame } = require('oni-save-parser');

const modsFile = JSON.parse(readFileSync('mods.json'))
console.log('parsing game... (this might take a while)')
const gameSave = parseSaveGame(readFileSync('world.sav').buffer, {
versionStrictness: 'none'
})

const gameActiveMods = gameSave.world.active_mods
const steamMods = modsFile.mods

const gameActiveModIds = gameActiveMods.map(m => m.id)
const steamModIds = steamMods.map(m => m.label.id)

const missingMods = gameActiveModIds.filter(id => steamModIds.indexOf(id) == -1).map(id => gameActiveMods[gameActiveMods.findIndex(m => m.id == id)])
if (missingMods.length > 0) {
console.log('The following mods are missing, please install/subscribe from Steam')
console.log(missingMods)
process.exit(1)
}

console.log(steamMods.map(m => m.label.title))

steamMods.sort((a, b) => {
const modIDA = a.label.id
const modIDB = b.label.id

const indexA = gameActiveModIds.indexOf(modIDA)
const indexB = gameActiveModIds.indexOf(modIDB)

// put active mods before inactive mods
if (indexA != -1 && indexB != -1) {
return indexA - indexB
}

if (indexA != -1) {
return -1
}

if (indexB != -1) {
return 1
}

// sort inactive mods by name
return a.label.title.toLowerCase().localeCompare(b.label.title.toLowerCase())
})

console.log(steamMods.map(m => m.label.title))

writeFileSync('mods.json', JSON.stringify(modsFile))

经手的一部分项目使用了rate.Limiter作为限流器, 但是使用了atomic.Value包裹. 猜测本意可能是为了多协程安全, 但实际上rate.Limiter本身就支持多协程访问. 最终导致了线上限流器行为与预期不符(大约超过限流2倍左右). 在更新限流器限流的时候, 应该使用 SetLimitSetLimitAt 来更新限流配置.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
func main() {
limiter := rate.NewLimiter(rate.Limit(50000), 10)
limiterStore.Store(limiter)
ctx := context.Background()

go func() {
for {
time.Sleep(time.Second * 1)
fmt.Printf("count: %v\n", count)
}
}()

go func() {
for {
time.Sleep(time.Second * 20)
newLimiter := rate.NewLimiter(rate.Limit(50000), 10)
limiterStore.Store(newLimiter)
}
}()

for i := 0; i < 1; i++ {
go func() {
for {
beginTs := time.Now()
for j := 0; j < 10000; j++ {
_ = limiterStore.Load().(*rate.Limiter).Wait(ctx)
}
duration := time.Since(beginTs)
fmt.Printf("dur: %v\n", duration.Nanoseconds())

count++
}
}()
}

time.Sleep(time.Hour * 24)
}

现象: Ctrl+Alt+M / 多显示器菜单切换无效, 选择新布局模式后屏闪并且仍然只使用一块显示器.

修复方式: 客户机打开服务: 停止 vm3dservice (VMware SVGA Helper Service) 将启动设置改为禁用即可.

VMware workstation pro 17 验证有效.

补充: 全屏模式下 Ctrl+Alt, Alt+Space, N 即可最小化当前正在运行的VM

参考

Multi-monitor still broken, does anybody have a solution / workaround ?

docker compose 模式部署的 gitea 在某次更新之后开始大量出现如下报错:

1
2
Feb 12 17:23:37 VM-0-16-ubuntu gitea_compose_giteadb_1[3927]: 2024-02-12  9:23:37 3103704 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'histogram' at position 10 to have type longblob, found type varbinary(255).
Feb 12 17:23:37 VM-0-16-ubuntu gitea_compose_giteadb_1[3927]: 2024-02-12 9:23:37 3103704 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'hist_type' at position 9 to have type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB','JSON_HB'), found type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB')

错误量非常巨大给syslog打崩了, 日志滚动的时候负载特别高会出现无法连接的问题. 经排查发现是mariadb升级的时候没有自动升级库表结构引起. 在 docker-compose.yml mariadb 中添加 MARIADB_AUTO_UPGRADE=1 环境变量再 docker-compose up -d 即可修复.

参考

Incorrect definition of table mysql.column_stats: expected column ‘hist_type’ … #25970

mariadb docker README

【大型催泪鬼畜】赵本山宋丹丹《白云黑土爱情故事》完整版MV首发!笑着笑着就哭了

Hoàng Read - The Magic Bomb (Questions I get asked) [Official Audio] 抽象手舞蹈BGM

ปูหนีบอีปิ (Poo Neep E-Pi) - พร จันทพร พอดีม่วน [OFFICIAL VIDEO] Lisa螃蟹舞BGM

Nee Jathaga Full Video Song - Yevadu Video Songs - Ram Charan, Allu Arjun, Shruti Hassan, Kajal Nee…

2NE1 - “I’m Busy (난 바빠) (Color Coded Lyrics Eng/Rom/Han/가사) bang bang bang bey bey bey

[MV] Lee Hyori(이효리) _ 10 Minutes

王不醒 - 佛说不配 (抖音热播DJ版) Phật Nói Không Xứng Đáng (Remix)『佛说不配,酒又下肚两杯,你有别人陪你不醉不归』【2k22抖音火流行歌曲推荐TikTok】

黄文文 - 财神殿里的上上签

Computer Chip Walking To Stayin’ Alive Synced to Music 原曲: Bee Gees - Stayin’ Alive (Official Music Video)

ERUPTION - ONE WAY TICKET / BONEY M - RASPUTIN / DJ DALI MIX / HQ HD

认清现实の小曲.mp3完整版素材 | The Amazing Digital Circus [End Orchestral Theme] | EPIC VERSION (Your New Home)

German Soldier’s Song - “Erika” (with English Subtitles) 希希哈哈的小曲

The Perfect Girl 绝命毒师meme曲

T-ARA(티아라) _ Sexy Love (Dance Ver. MV) B站舞蹈区BGM Sexy Love

Project Zomboid Remastered OST - Main Theme PZ主题曲

我知道你对我最好 | 《小丑の小曲》1.3x蚊子版 | 勵陽 - 我知道你『我知道你 在我世界最重要』【動態歌詞Lyrics】

往期优秀作品推荐

2023年9-10月

问题现象

1
2
sudo podman create network test
sudo podman network ls

报错提示

1
WARN[0000] Error validating CNI config file /etc/cni/net.d/test.conflist: [plugin bridge does not support config version "1.0.0" plugin portmap does not support config version "1.0.0" plugin firewall does not support config version "1.0.0" plugin tuning does not support config version "1.0.0"]

解决方案

看起来是Ubuntu2204的已知问题, 需要手动安装最新版本的containernetworking-plugins

  1. 打开下载页面, 下载最新的deb包

  2. sudo dpkg -i containernetworking-plugins_1.1.1+ds1-3build1_amd64.deb

参考

Ubuntu 22.04.1 LTS libpod (package podman 3.4.4+ds1-1ubuntu1.22.04.1): broken network functionality for CNI plugins

Podman automatically sets cniVersion 1.0.0 instead of 0.4.0

Error validating CNI config file and network errors in rootful podman #14189

需要两个配置文件

消费远端消息 mirrormaker-consumer.config (注: remote-kafka 需要在 /etc/hosts 里添加配置, 下同)

1
2
bootstrap.servers=remote-kafka:9092
group.id=mirrormaker-mirror-1

投递到本地的Kafka集群 mirrormaker-producer.config

1
2
bootstrap.servers=local-kafka:9092
acks=1

启动 Kafka Mirror Maker

1
2
#!/bin/bash
kafka-mirror-maker --consumer.config mirrormaker-consumer.config --producer.config mirrormaker-producer.config --include 'topic1|topic2'

mirror-maker虽然不被confluent看重(可能是因为他们有专门的同步工具), 不过还是非常好用的. 在线上实际跑了将近三个月的时间, 其自带的重试功能可以保证同步意外断开之后无人值守地自动恢复.

参考

retention.ms - Kafka topic configuration reference | Confluent Documentation 配置消息过期时间

Docker Configuration Parameters for Confluent Platform

Migrating clusters using Apache Kafka’s MirrorMaker

Kafka MirrorMaker

Kafka mirroring (MirrorMaker)