Mattermost Operations Runbook

Operational reference for the Mattermost Team Edition deployment that powers Equa team messaging.

Infrastructure Overview

Component	Details
Platform	Railway (Hobby plan)
Project	`equa-mattermost`
Project ID	`56d9cfed-2b45-4452-a6f8-0685ab8f7291`
Environment	`production` (`751529d4-d056-49c9-b323-6cef763a9f0a`)

Services

Service	Image	Service ID	Internal Domain
Mattermost	`mattermost/mattermost-team-edition:release-10.4`	`d4b65ddc-6a7c-4a5a-a357-0fb37674f39a`	`mattermost.railway.internal`
PostgreSQL	`postgres:16-alpine`	`6ae96c74-d62c-4957-85f2-2f479768268a`	`postgres.railway.internal`

Domains

Domain	Type	Target
`mattermost-production.up.railway.app`	Railway service domain	Mattermost (port 8065)
`chat.equa.cc`	Custom domain (CNAME)	`ek6lcxl4.up.railway.app`

Volumes

Volume	Mount Path	Service
`postgres-volume`	`/var/lib/postgresql/data`	PostgreSQL
`mattermost-volume`	`/mattermost/data`	Mattermost

Credentials Inventory

All credentials are stored as Railway environment variables. Never commit credentials to source control.

Mattermost Admin Account

Property	Value
Username	`equa-admin`
Email	`shawn@owenent.com`
Role	`system_admin`
Location	Railway dashboard or local password manager

Admin Bot Account

Property	Value
Username	`equa-admin-bot`
Display Name	Equa Platform
User ID	`cqh9a9u8wi8i5neqwkfxdntq7a`
Token location	`MATTERMOST_ADMIN_BOT_TOKEN` env var on equa-server Railway project

PostgreSQL

Property	Value
Database	`mattermost`
User	`mattermost`
Password location	`POSTGRES_PASSWORD` env var on equa-mattermost Railway project
Host (internal)	`postgres.railway.internal:5432`

Environment Variables on equa-server

Variable	Value	Project
`MATTERMOST_URL`	`https://chat.equa.cc`	equa-server-so
`MATTERMOST_ADMIN_BOT_TOKEN`	(stored in Railway)	equa-server-so

DNS Configuration

DNS is managed in Google Cloud DNS, project equa-production, zone equa-cc-zone.

Record	Type	TTL	Value
`chat.equa.cc`	CNAME	300	`ek6lcxl4.up.railway.app`
`_railway-verify.chat.equa.cc`	TXT	300	`railway-verify=2463d41c43c8bfab337b245fbcac8b9264790df52a78c45a2a715fa003963a98`

Modifying DNS

# Requires gcloud auth with access to equa-production project
gcloud auth login
gcloud dns record-sets update chat.equa.cc. \
  --zone=equa-cc-zone \
  --type=CNAME \
  --ttl=300 \
  --rrdatas="NEW_TARGET." \
  --project=equa-production

Monitoring

Health Check

curl https://chat.equa.cc/api/v4/system/ping
# Expected: {"status":"OK", ...}

Admin Token Verification

curl -H "Authorization: Bearer $MATTERMOST_ADMIN_BOT_TOKEN" \
  https://chat.equa.cc/api/v4/users/me
# Expected: {"username":"equa-admin","roles":"system_admin system_user", ...}

Despite the env var name MATTERMOST_ADMIN_BOT_TOKEN, this is a personal access token belonging to the equa-admin regular user, not the bot account.

Check Teams

curl -H "Authorization: Bearer $MATTERMOST_ADMIN_BOT_TOKEN" \
  https://chat.equa.cc/api/v4/teams

Common Operations

Restart Mattermost

Via Railway CLI:

railway link --project equa-mattermost
railway service mattermost
railway redeploy

Via Railway GraphQL API:

curl -X POST https://backboard.railway.com/graphql/v2 \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $RAILWAY_TOKEN" \
  -d '{"query":"mutation { serviceInstanceRedeploy(environmentId: \"751529d4-d056-49c9-b323-6cef763a9f0a\", serviceId: \"d4b65ddc-6a7c-4a5a-a357-0fb37674f39a\") }"}'

View Deployment Logs

railway link --project equa-mattermost
railway service mattermost
railway logs

Update Mattermost Version

Update the Docker image tag in Railway:

# Via GraphQL API - update the service source image
curl -X POST https://backboard.railway.com/graphql/v2 \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $RAILWAY_TOKEN" \
  -d '{"query":"mutation { serviceUpdate(id: \"d4b65ddc-6a7c-4a5a-a357-0fb37674f39a\", input: { source: { image: \"mattermost/mattermost-team-edition:release-10.5\" } }) { id } }"}'

Then redeploy the service.

Rotate Admin Token

MATTERMOST_ADMIN_BOT_TOKEN is a personal access token belonging to the equa-admin regular user, not the bot account. Bot tokens cannot create PATs for other users due to a Mattermost platform limitation.

Standard rotation (zero downtime):

Log into Mattermost as equa-admin (shawn@owenent.com)
Go to Profile > Security > Personal Access Tokens
Click Create Token, give it a description (e.g., Equa session 2026-03)
Copy the new token value
In Railway (equa-server-so project), update MATTERMOST_ADMIN_BOT_TOKEN to the new token
Wait for equa-server to redeploy
Verify: POST /api/v1/mattermost/session succeeds
Revoke the old token in Mattermost (Profile > Security > Personal Access Tokens)

The old token continues working until explicitly revoked, so there is no downtime window between steps 5 and 8. Emergency recovery (admin locked out):

Enable the TCP proxy (see TCP Proxy Management below)
Connect to the Mattermost database:

psql -h shinkansen.proxy.rlwy.net -p 27531 -U mattermost -d mattermost

Reset the admin password:

-- Generate hash with: node -e "console.log(require('bcryptjs').hashSync('NewPassword123!', 10))"
UPDATE users SET password = '$2a$10$...' WHERE username = 'equa-admin';

Log in with the new password, create a new PAT
Update MATTERMOST_ADMIN_BOT_TOKEN in Railway
Disable the TCP proxy

Full Member Sync

If member state is out of sync between Equa and Mattermost:

# Via equa-server API (requires admin session cookie)
curl -X POST https://equa-server-so-production.up.railway.app/api/v1/mattermost/sync-org/{orgId} \
  -H "Cookie: session=..."

Mattermost Environment Variables

Key configuration on the Mattermost Railway service:

Variable	Value	Purpose
`MM_SQLSETTINGS_DRIVERNAME`	`postgres`	Database driver
`MM_SQLSETTINGS_DATASOURCE`	`postgres://mattermost:***@postgres.railway.internal:5432/mattermost?sslmode=disable`	Database connection
`MM_SERVICESETTINGS_SITEURL`	`https://chat.equa.cc`	Public site URL
`MM_SERVICESETTINGS_LISTENADDRESS`	`:8065`	Listen port
`MM_SERVICESETTINGS_ENABLEPERSONALACCESSTOKENS`	`true`	Required for API-provisioned auth
`MM_SERVICESETTINGS_ENABLEBOTACCOUNTCREATION`	`true`	Required for admin bot
`MM_SERVICESETTINGS_ENABLEUSERACCESSTOKENS`	`true`	Required for user provisioning
`MM_SERVICESETTINGS_ALLOWCORSFROM`	`https://app.equa.cc`	CORS for cross-origin API requests (does not control iframe embedding — see Nginx Reverse Proxy section)
`MM_TEAMSETTINGS_ENABLEOPENSERVER`	`false`	Prevent public registration
`MM_EMAILSETTINGS_ENABLESIGNUPWITHEMAIL`	`false`	Prevent email signup (users provisioned via Equa only)
`PORT`	`8065`	Railway port binding

Nginx Reverse Proxy

Mattermost returns X-Frame-Options: SAMEORIGIN by default, which blocks cross-origin iframe embedding. Since equa-web embeds Mattermost in an iframe, an nginx reverse proxy sits in front of the Mattermost service to strip this header and add permissive Content-Security-Policy: frame-ancestors rules.

Architecture

equa-web (iframe src=chat.equa.cc)
  → chat.equa.cc (DNS CNAME)
    → Nginx Reverse Proxy (Railway service, port 8080)
      → Mattermost (mattermost.railway.internal:8065)

Configuration

Source: equa-server/infra/mattermost-proxy/

File	Purpose
`Dockerfile`	nginx:alpine with template-based config
`nginx.conf`	Strips `X-Frame-Options`, adds `frame-ancestors`, proxies to Mattermost
`README.md`	Deployment instructions

Environment Variables

Variable	Description	Example
`MATTERMOST_UPSTREAM`	Internal Mattermost URL	`http://mattermost.railway.internal:8065`

Deployment

In the equa-mattermost Railway project, create a new service from equa-server/infra/mattermost-proxy/
Set MATTERMOST_UPSTREAM to the internal Mattermost URL
Move the chat.equa.cc custom domain from the Mattermost service to the proxy service
The proxy listens on port 8080 (auto-detected from the Dockerfile EXPOSE)

Verification

# Confirm X-Frame-Options is stripped
curl -I https://chat.equa.cc | grep -i x-frame-options
# Should return nothing

# Confirm frame-ancestors is set
curl -I https://chat.equa.cc | grep -i content-security-policy
# Should return: frame-ancestors 'self' https://equa-web-so-production.up.railway.app https://app.equa.cc

If the proxy is bypassed or removed, the messaging iframe will show a blank page. The X-Frame-Options header is set by Mattermost itself and cannot be disabled via Mattermost configuration.

TCP Proxy Management

The Mattermost PostgreSQL database has a TCP proxy for emergency direct access.

Property	Value
Host	`shinkansen.proxy.rlwy.net`
Port	`27531`
User	`mattermost`
Database	`mattermost`
SSL	Disabled

When to Enable

Password resets for locked-out admin accounts
Direct role modifications not possible via the API
Database migration troubleshooting
One-time data corrections

When to Disable

Disable the TCP proxy whenever it is not actively needed for admin operations.

How to Toggle

Open the equa-mattermost Railway project
Navigate to the PostgreSQL service
Go to Settings > Networking > Public Networking
Toggle the TCP proxy off (disable) or on (enable)

Risk Assessment

State	Risk	Mitigation
Enabled	Database reachable from public internet	Protected by database credentials; but credentials could be leaked via env vars, logs, or config drift
Disabled	No remote DB access for emergency ops	Use Railway built-in query interface or re-enable temporarily

Recommendation: Keep disabled by default. Enable only for specific admin tasks, then disable immediately after.

Scaling Considerations

Mattermost Team Edition on Railway Hobby plan:

Resource	Current	Limit	Action if exceeded
RAM	~512MB	Hobby plan limits	Upgrade Railway plan or migrate to dedicated VPS
Storage	Volume-backed	Hobby plan limits	Monitor via Railway dashboard
Connections	Low (single org testing)	PostgreSQL connection limits	Add connection pooling if needed

For production scale (50+ concurrent users), consider:

Upgrading to Railway Pro plan
Adding Redis for session caching (MM_CACHEETTINGS_CACHETYPE=redis)
Configuring S3 for file storage (MM_FILESETTINGS_DRIVERNAME=amazons3)
Setting up log aggregation

​Mattermost Operations Runbook

​Infrastructure Overview

​Services

​Domains

​Volumes

​Credentials Inventory

​Mattermost Admin Account

​Admin Bot Account

​PostgreSQL

​Environment Variables on equa-server

​DNS Configuration

​Modifying DNS

​Monitoring

​Health Check

​Admin Token Verification

​Check Teams

​Common Operations

​Restart Mattermost

​View Deployment Logs

​Update Mattermost Version

​Rotate Admin Token

​Full Member Sync

​Mattermost Environment Variables

​Nginx Reverse Proxy

​Architecture

​Configuration

​Environment Variables

​Deployment

​Verification

​TCP Proxy Management

​When to Enable

​When to Disable

​How to Toggle

​Risk Assessment

​Scaling Considerations

Mattermost Operations Runbook

Infrastructure Overview

Services

Domains

Volumes

Credentials Inventory

Mattermost Admin Account

Admin Bot Account

PostgreSQL

Environment Variables on equa-server

DNS Configuration

Modifying DNS

Monitoring

Health Check

Admin Token Verification

Check Teams

Common Operations

Restart Mattermost

View Deployment Logs

Update Mattermost Version

Rotate Admin Token

Full Member Sync

Mattermost Environment Variables

Nginx Reverse Proxy

Architecture

Configuration

Environment Variables

Deployment

Verification

TCP Proxy Management

When to Enable

When to Disable

How to Toggle

Risk Assessment

Scaling Considerations