blog-astro/docs/deploy-k3s.md

# blog_astro K3s Deployment

Last updated: 2026-04-06

## Overview

This project is deployed to the `notion-stack` namespace in the existing K3s cluster.

Current traffic topology:

- Public entry: Traefik `IngressRoute` for `nvme0n1p.dev`
- Public service target: `notion-blog-api:3000`
- Frontend upstream inside cluster: `http://blog-astro:3000`
- Frontend runtime: Astro SSR in a container listening on `3000`
- Preferred build host: `root@10.66.66.3`
- Fallback build host: `root@156.239.238.24`

Important: the public Ingress does **not** point directly to `blog-astro`. The site is exposed through `notion-blog-api`, which proxies the frontend via `FRONTEND_UPSTREAM=http://blog-astro:3000`.

## Why Build Remotely

Local development may happen on a non-amd64 machine, while the K3s node running this workload is amd64. To avoid cross-architecture image issues, build the image on the amd64 Ubuntu host, then import it into K3s containerd on that same host.

The frontend workload is pinned to the `ubuntu` node.

## Current Kubernetes Layout

- Namespace: `notion-stack`
- Deployment: `blog-astro`
- Service: `blog-astro`
- Container port: `3000`
- Image pull mode: `Never`
- Node selector: `kubernetes.io/hostname=ubuntu`

Useful checks:

```bash
ssh root@10.66.66.3 'kubectl -n notion-stack get deploy,svc,pods -l app=blog-astro -o wide'
ssh root@10.66.66.3 'kubectl -n notion-stack get ingressroute notion-stack -o yaml'
ssh root@10.66.66.3 'kubectl -n notion-stack get deploy notion-blog-api -o yaml'
```

## Deployment Procedure

### 1. Sync the current working tree to the remote build host

This pushes the current local state, including uncommitted changes.

```bash
rsync -az --delete \
  --exclude '.git' \
  --exclude 'node_modules' \
  --exclude 'dist' \
  --exclude '.astro' \
  --exclude '.vercel' \
  --exclude '.DS_Store' \
  ./ root@10.66.66.3:/root/deploy/blog_astro/
```

If `10.66.66.3` is unavailable, use:

```bash
rsync -az --delete \
  --exclude '.git' \
  --exclude 'node_modules' \
  --exclude 'dist' \
  --exclude '.astro' \
  --exclude '.vercel' \
  --exclude '.DS_Store' \
  ./ root@156.239.238.24:/root/deploy/blog_astro/
```

### 2. Build a new uniquely tagged image on the remote host

Do not reuse `blog-astro:deploy` for the build itself. Use a timestamped tag first so the rollout target is explicit.

Example:

```bash
ssh root@10.66.66.3 '
  cd /root/deploy/blog_astro &&
  docker build -t blog-astro:deploy-20260406-1251 .
'
```

If the SSH session is unstable, run the build in the background and log to a file:

```bash
ssh root@10.66.66.3 '
  cd /root/deploy/blog_astro &&
  rm -f build.log build.ok build.fail &&
  nohup sh -lc '\''docker build -t blog-astro:deploy-20260406-1251 . > build.log 2>&1 && echo ok > build.ok || echo fail > build.fail'\'' >/dev/null 2>&1 &
'

ssh root@10.66.66.3 '
  cd /root/deploy/blog_astro &&
  if [ -f build.ok ]; then
    echo BUILD_OK
  elif [ -f build.fail ]; then
    echo BUILD_FAIL
  else
    echo BUILD_RUNNING
  fi &&
  tail -n 40 build.log
'
```

### 3. Import the built image into K3s containerd

The Docker daemon image is not automatically visible to K3s. Import it explicitly:

```bash
ssh root@10.66.66.3 '
  docker save blog-astro:deploy-20260406-1251 | ctr -n k8s.io images import -
'
```

Verify the imported image exists:

```bash
ssh root@10.66.66.3 '
  ctr -n k8s.io images ls | grep "blog-astro.*deploy-20260406-1251"
'
```

### 4. Update the Deployment to the new image

```bash
ssh root@10.66.66.3 '
  kubectl -n notion-stack set image deployment/blog-astro \
    blog-astro=blog-astro:deploy-20260406-1251 &&
  kubectl -n notion-stack rollout status deployment/blog-astro --timeout=180s
'
```

Inspect the result:

```bash
ssh root@10.66.66.3 '
  kubectl -n notion-stack get deploy blog-astro -o wide &&
  kubectl -n notion-stack get pods -l app=blog-astro -o wide
'
```

## Verification

### Verify the frontend service directly

```bash
ssh root@10.66.66.3 '
  curl -sS -o /tmp/blog_home.html -w "%{http_code}\n" http://10.43.98.176:3000/ &&
  grep -o "<title>[^<]*</title>" /tmp/blog_home.html | head -n 1
'
```

### Verify through the public entrypoint

This confirms Traefik -> `notion-blog-api` -> `blog-astro` is still correct.

```bash
ssh root@10.66.66.3 '
  curl -sS -H "Host: nvme0n1p.dev" \
    -o /tmp/ext_home.html \
    -w "%{http_code}\n" \
    http://127.0.0.1:31349/ &&
  grep -o "<title>[^<]*</title>" /tmp/ext_home.html | head -n 1
'
```

### 2026-04-06 verification example

- New image: `blog-astro:deploy-20260406-1251`
- New pod: `blog-astro-58f9c8cf4c-bj9zv`
- Frontend service `/`: `200`, title `Home | 溴化锂的笔记本`
- Frontend service `/blog/apfs-sparsebundle-linux`: `200`
- Public entry `/`: `200`, title `Home | 溴化锂的笔记本`
- Public entry `/blog/apfs-sparsebundle-linux`: `200`

## Rollback

List old ReplicaSets and images:

```bash
ssh root@10.66.66.3 '
  kubectl -n notion-stack get rs -l app=blog-astro &&
  ctr -n k8s.io images ls | grep blog-astro
'
```

Rollback by switching the Deployment image back to the previous known-good tag:

```bash
ssh root@10.66.66.3 '
  kubectl -n notion-stack set image deployment/blog-astro \
    blog-astro=blog-astro:deploy
'
```

If the previous version also used a unique tag, set that exact tag instead.

## Troubleshooting

### SSH connects on port 22 but does not return a banner

Symptom:

- `nc -vz <host> 22` succeeds
- `ssh` hangs or fails during banner exchange

Likely cause:

- `sshd` is overloaded or limiting new unauthenticated sessions

Mitigation:

- Avoid many parallel SSH probes
- Prefer a single session
- Run long builds with `nohup` and inspect `build.log`

### New Docker image is built but Pods still use the old one

Cause:

- K3s uses containerd, and `imagePullPolicy: Never` means Kubernetes only sees images already imported into containerd

Fix:

```bash
ssh root@10.66.66.3 '
  docker save blog-astro:<tag> | ctr -n k8s.io images import -
'
```

### Public site still works even though Ingress does not point to `blog-astro`

This is expected. Public traffic goes:

`Traefik -> notion-blog-api -> blog-astro`

So frontend deployments normally do not require Ingress changes.