Production Server Governance & Remote Access to an Internal Gitea

The problem: it runs, but nobody can take it over

The state I inherited is common: a few people hold root, accounts were added verbally, releases happen when someone says “restart it”, and incidents are reconstructed from group chat. A system like that is not unusable — it is un-handoverable, because all the operating knowledge hangs off specific people, and the risk surfaces the moment one of them leaves.

So the goal was not “install a few more services” but to write three things into the system: who may enter, what they may do, and how it gets rescued.

Accounts: turn provisioning and revocation into scripts

Accounts are the root of all permissions and the easiest thing to bypass by hand. The rule here: only the script may create an account; a manual useradd is treated as an incident.

Stage	Action	Output
Provision	Collect public key → create the system account in sysops → install the key with a hardening prefix → generate a random sudo password → create the DB user → write the ledger	A deliverable provisioning result and one ledger row
Use	Home directory 700/700/600; keys carry no-agent-forwarding and friends; sudo requires a password	Least privilege, fully traceable
Revoke	Remove keys, lock the account, drop the DB user, look for orphaned files and config references, reconcile cloud RAM and ActionTrail	A revocation checklist, confirmed item by item

The ledger (accounts.tsv, 640 root:sysops) records accounts and key fingerprints and never a password — passwords travel through a password manager or a one-time link, never as plain text in chat. “Who executed what, as whom, and when” is answered by the sudo audit log, not by recollection.

Security baseline: exposed by default means exposed

Security is not a switch; it is a set of defaults. On this host the default is “close every entrance you do not need”:

• SSH: password auth off, keyboard-interactive off, three attempts maximum, only the sysops group and root, root by key only.
• Databases: MySQL bound to 127.0.0.1 with root over the socket; Redis bound to loopback. Security groups never open 3306 / 33060 / 6379.
• Self-checks: db-exposure-check verifies binds and exposed ports daily; user-audit emits a weekly snapshot of accounts, key fingerprints, password age and the effective sshd configuration.
• Cloud side: the RAM sub-account gets read-only plus session management — no RunCommand, no snapshots, no password reset — and ActionTrail keeps the audit trail.

Baseline snapshots (sshd config, sudoers, account and key fingerprints, listening ports) are taken before any change, so the whole thing can be rolled back — decide how to retreat before you advance.

Release and rollback: one command that knows how to fail

Releasing is where things break, so it is compressed into one command that carries its own way back:

deploy-backend <jar>     backup → replace → restart → probe → automatic rollback on failure
deploy-frontend <dist>   tar backup → clean unpack → validate index.html

The rules around it are equally explicit: release off-peak; back up before any SQL change; afterwards check the service state, the last two minutes of logs for ERROR, and a few key pages; keep disk headroom (two or three backups, no more). Nobody improvises a rollback — the previous version is already in *.bak-*, and the same script restores it.

Remote work: an internal Gitea with zero public ports

The internal Gitea was reachable only from the office subnet, so colleagues at home could not push. The usual answers — open a public port, or stand up a VPN — both enlarge the attack surface.

Version one · Tailscale: put the Gitea host and each colleague’s laptop into one encrypted tailnet and reach it over a virtual IP. No public port, no port forwarding, no VPN appliance; access is granted per device, and tailscale ping tells you whether the path is direct or relayed.

Version two · self-hosted Headscale: the managed Tailscale depends on a third-party account and caps a tailnet at three users — a hard constraint for a team. So Headscale became the control server: it only coordinates (registration, address assignment, key exchange, ACLs), while traffic stays peer-to-peer and falls back to DERP relaying only when a direct path fails. The Gitea host advertises the internal range as a subnet router, so a colleague at home can keep using the same internal address with almost no client change.

The trade is stated up front: self-hosting buys control and unlimited seats at the cost of maintaining the control server, its certificates and its upgrades.

The document is the deliverable

What actually ships is not a configured machine but ten documents you can follow:

• The Admin Manual: account baseline, provisioning and revocation, database management, release and rollback, audit and inspection, incident response, cloud-side controls, known pitfalls.
• Nine Colleague Handbooks: request an account → self-check and setup → quick reference → server rules → common commands → deploy code → database operations → error lookup → when to call the admin.

They are deliberately de-personalised: commands are labelled with the machine they run on, values are placeholders, and every step states what you should expect to see. Every claim comes from a real run — for instance, “a normal colleague account cannot read root’s journal, the real application logs live in the app directory” was discovered while debugging, not assumed.

Outcome

• New colleagues went from “wait for the admin” to self-service: request, self-check, deploy, connect, troubleshoot — each with a documented path.
• Releases went from “operate from memory” to “one command with automatic rollback”, so a failure no longer needs an on-the-spot decision.
• Remote work went from “open a port / run a VPN” to “a device inside the tailnet or nothing”, with zero public exposure.
• Accounts went from “anyone can add one” to “only the script can”, with revocation that has a checklist, an audit trail and something to reconcile against.