> ## Documentation Index
> Fetch the complete documentation index at: https://docs.equa.cc/llms.txt
> Use this file to discover all available pages before exploring further.

# Mattermost Operations Runbook

> Infrastructure, credentials, DNS, and operational procedures for the Mattermost deployment

# Mattermost Operations Runbook

Operational reference for the Mattermost Team Edition deployment that powers Equa team messaging.

***

## Infrastructure Overview

| Component   | Details                                               |
| ----------- | ----------------------------------------------------- |
| Platform    | Railway (Hobby plan)                                  |
| Project     | `equa-mattermost`                                     |
| Project ID  | `56d9cfed-2b45-4452-a6f8-0685ab8f7291`                |
| Environment | `production` (`751529d4-d056-49c9-b323-6cef763a9f0a`) |

### Services

| Service    | Image                                             | Service ID                             | Internal Domain               |
| ---------- | ------------------------------------------------- | -------------------------------------- | ----------------------------- |
| Mattermost | `mattermost/mattermost-team-edition:release-10.4` | `d4b65ddc-6a7c-4a5a-a357-0fb37674f39a` | `mattermost.railway.internal` |
| PostgreSQL | `postgres:16-alpine`                              | `6ae96c74-d62c-4957-85f2-2f479768268a` | `postgres.railway.internal`   |

### Domains

| Domain                                 | Type                   | Target                    |
| -------------------------------------- | ---------------------- | ------------------------- |
| `mattermost-production.up.railway.app` | Railway service domain | Mattermost (port 8065)    |
| `chat.equa.cc`                         | Custom domain (CNAME)  | `ek6lcxl4.up.railway.app` |

### Volumes

| Volume              | Mount Path                 | Service    |
| ------------------- | -------------------------- | ---------- |
| `postgres-volume`   | `/var/lib/postgresql/data` | PostgreSQL |
| `mattermost-volume` | `/mattermost/data`         | Mattermost |

***

## Credentials Inventory

<Warning>All credentials are stored as Railway environment variables. Never commit credentials to source control.</Warning>

### Mattermost Admin Account

| Property | Value                                       |
| -------- | ------------------------------------------- |
| Username | `equa-admin`                                |
| Email    | `shawn@owenent.com`                         |
| Role     | `system_admin`                              |
| Location | Railway dashboard or local password manager |

### Admin Bot Account

| Property       | Value                                                               |
| -------------- | ------------------------------------------------------------------- |
| Username       | `equa-admin-bot`                                                    |
| Display Name   | Equa Platform                                                       |
| User ID        | `cqh9a9u8wi8i5neqwkfxdntq7a`                                        |
| Token location | `MATTERMOST_ADMIN_BOT_TOKEN` env var on equa-server Railway project |

### PostgreSQL

| Property          | Value                                                          |
| ----------------- | -------------------------------------------------------------- |
| Database          | `mattermost`                                                   |
| User              | `mattermost`                                                   |
| Password location | `POSTGRES_PASSWORD` env var on equa-mattermost Railway project |
| Host (internal)   | `postgres.railway.internal:5432`                               |

### Environment Variables on equa-server

| Variable                     | Value                  | Project        |
| ---------------------------- | ---------------------- | -------------- |
| `MATTERMOST_URL`             | `https://chat.equa.cc` | equa-server-so |
| `MATTERMOST_ADMIN_BOT_TOKEN` | (stored in Railway)    | equa-server-so |

***

## DNS Configuration

DNS is managed in Google Cloud DNS, project `equa-production`, zone `equa-cc-zone`.

| Record                         | Type  | TTL | Value                                                                             |
| ------------------------------ | ----- | --- | --------------------------------------------------------------------------------- |
| `chat.equa.cc`                 | CNAME | 300 | `ek6lcxl4.up.railway.app`                                                         |
| `_railway-verify.chat.equa.cc` | TXT   | 300 | `railway-verify=2463d41c43c8bfab337b245fbcac8b9264790df52a78c45a2a715fa003963a98` |

### Modifying DNS

```bash theme={null}
# Requires gcloud auth with access to equa-production project
gcloud auth login
gcloud dns record-sets update chat.equa.cc. \
  --zone=equa-cc-zone \
  --type=CNAME \
  --ttl=300 \
  --rrdatas="NEW_TARGET." \
  --project=equa-production
```

***

## Monitoring

### Health Check

```bash theme={null}
curl https://chat.equa.cc/api/v4/system/ping
# Expected: {"status":"OK", ...}
```

### Admin Token Verification

```bash theme={null}
curl -H "Authorization: Bearer $MATTERMOST_ADMIN_BOT_TOKEN" \
  https://chat.equa.cc/api/v4/users/me
# Expected: {"username":"equa-admin","roles":"system_admin system_user", ...}
```

<Note>Despite the env var name `MATTERMOST_ADMIN_BOT_TOKEN`, this is a personal access token belonging to the `equa-admin` regular user, not the bot account.</Note>

### Check Teams

```bash theme={null}
curl -H "Authorization: Bearer $MATTERMOST_ADMIN_BOT_TOKEN" \
  https://chat.equa.cc/api/v4/teams
```

***

## Common Operations

### Restart Mattermost

Via Railway CLI:

```bash theme={null}
railway link --project equa-mattermost
railway service mattermost
railway redeploy
```

Via Railway GraphQL API:

```bash theme={null}
curl -X POST https://backboard.railway.com/graphql/v2 \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $RAILWAY_TOKEN" \
  -d '{"query":"mutation { serviceInstanceRedeploy(environmentId: \"751529d4-d056-49c9-b323-6cef763a9f0a\", serviceId: \"d4b65ddc-6a7c-4a5a-a357-0fb37674f39a\") }"}'
```

### View Deployment Logs

```bash theme={null}
railway link --project equa-mattermost
railway service mattermost
railway logs
```

### Update Mattermost Version

Update the Docker image tag in Railway:

```bash theme={null}
# Via GraphQL API - update the service source image
curl -X POST https://backboard.railway.com/graphql/v2 \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $RAILWAY_TOKEN" \
  -d '{"query":"mutation { serviceUpdate(id: \"d4b65ddc-6a7c-4a5a-a357-0fb37674f39a\", input: { source: { image: \"mattermost/mattermost-team-edition:release-10.5\" } }) { id } }"}'
```

Then redeploy the service.

### Rotate Admin Token

<Warning>`MATTERMOST_ADMIN_BOT_TOKEN` is a personal access token belonging to the `equa-admin` regular user, not the bot account. Bot tokens cannot create PATs for other users due to a Mattermost platform limitation.</Warning>

**Standard rotation (zero downtime):**

1. Log into Mattermost as `equa-admin` (`shawn@owenent.com`)
2. Go to **Profile > Security > Personal Access Tokens**
3. Click **Create Token**, give it a description (e.g., `Equa session 2026-03`)
4. Copy the new token value
5. In Railway (`equa-server-so` project), update `MATTERMOST_ADMIN_BOT_TOKEN` to the new token
6. Wait for equa-server to redeploy
7. Verify: `POST /api/v1/mattermost/session` succeeds
8. Revoke the old token in Mattermost (Profile > Security > Personal Access Tokens)

The old token continues working until explicitly revoked, so there is no downtime window between steps 5 and 8.

**Emergency recovery (admin locked out):**

1. Enable the TCP proxy (see TCP Proxy Management below)
2. Connect to the Mattermost database:

```bash theme={null}
psql -h shinkansen.proxy.rlwy.net -p 27531 -U mattermost -d mattermost
```

3. Reset the admin password:

```sql theme={null}
-- Generate hash with: node -e "console.log(require('bcryptjs').hashSync('NewPassword123!', 10))"
UPDATE users SET password = '$2a$10$...' WHERE username = 'equa-admin';
```

4. Log in with the new password, create a new PAT
5. Update `MATTERMOST_ADMIN_BOT_TOKEN` in Railway
6. Disable the TCP proxy

### Full Member Sync

If member state is out of sync between Equa and Mattermost:

```bash theme={null}
# Via equa-server API (requires admin session cookie)
curl -X POST https://equa-server-so-production.up.railway.app/api/v1/mattermost/sync-org/{orgId} \
  -H "Cookie: session=..."
```

***

## Mattermost Environment Variables

Key configuration on the Mattermost Railway service:

| Variable                                        | Value                                                                                 | Purpose                                                                                                       |
| ----------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `MM_SQLSETTINGS_DRIVERNAME`                     | `postgres`                                                                            | Database driver                                                                                               |
| `MM_SQLSETTINGS_DATASOURCE`                     | `postgres://mattermost:***@postgres.railway.internal:5432/mattermost?sslmode=disable` | Database connection                                                                                           |
| `MM_SERVICESETTINGS_SITEURL`                    | `https://chat.equa.cc`                                                                | Public site URL                                                                                               |
| `MM_SERVICESETTINGS_LISTENADDRESS`              | `:8065`                                                                               | Listen port                                                                                                   |
| `MM_SERVICESETTINGS_ENABLEPERSONALACCESSTOKENS` | `true`                                                                                | Required for API-provisioned auth                                                                             |
| `MM_SERVICESETTINGS_ENABLEBOTACCOUNTCREATION`   | `true`                                                                                | Required for admin bot                                                                                        |
| `MM_SERVICESETTINGS_ENABLEUSERACCESSTOKENS`     | `true`                                                                                | Required for user provisioning                                                                                |
| `MM_SERVICESETTINGS_ALLOWCORSFROM`              | `https://app.equa.cc`                                                                 | CORS for cross-origin API requests (does **not** control iframe embedding -- see Nginx Reverse Proxy section) |
| `MM_TEAMSETTINGS_ENABLEOPENSERVER`              | `false`                                                                               | Prevent public registration                                                                                   |
| `MM_EMAILSETTINGS_ENABLESIGNUPWITHEMAIL`        | `false`                                                                               | Prevent email signup (users provisioned via Equa only)                                                        |
| `PORT`                                          | `8065`                                                                                | Railway port binding                                                                                          |

***

## Nginx Reverse Proxy

Mattermost returns `X-Frame-Options: SAMEORIGIN` by default, which blocks cross-origin iframe embedding. Since equa-web embeds Mattermost in an iframe, an nginx reverse proxy sits in front of the Mattermost service to strip this header and add permissive `Content-Security-Policy: frame-ancestors` rules.

### Architecture

```
equa-web (iframe src=chat.equa.cc)
  → chat.equa.cc (DNS CNAME)
    → Nginx Reverse Proxy (Railway service, port 8080)
      → Mattermost (mattermost.railway.internal:8065)
```

### Configuration

Source: `equa-server/infra/mattermost-proxy/`

| File         | Purpose                                                                 |
| ------------ | ----------------------------------------------------------------------- |
| `Dockerfile` | nginx:alpine with template-based config                                 |
| `nginx.conf` | Strips `X-Frame-Options`, adds `frame-ancestors`, proxies to Mattermost |
| `README.md`  | Deployment instructions                                                 |

### Environment Variables

| Variable              | Description             | Example                                   |
| --------------------- | ----------------------- | ----------------------------------------- |
| `MATTERMOST_UPSTREAM` | Internal Mattermost URL | `http://mattermost.railway.internal:8065` |

### Deployment

1. In the **equa-mattermost** Railway project, create a new service from `equa-server/infra/mattermost-proxy/`
2. Set `MATTERMOST_UPSTREAM` to the internal Mattermost URL
3. Move the `chat.equa.cc` custom domain from the Mattermost service to the proxy service
4. The proxy listens on port 8080 (auto-detected from the Dockerfile `EXPOSE`)

### Verification

```bash theme={null}
# Confirm X-Frame-Options is stripped
curl -I https://chat.equa.cc | grep -i x-frame-options
# Should return nothing

# Confirm frame-ancestors is set
curl -I https://chat.equa.cc | grep -i content-security-policy
# Should return: frame-ancestors 'self' https://equa-web-so-production.up.railway.app https://app.equa.cc
```

<Warning>If the proxy is bypassed or removed, the messaging iframe will show a blank page. The `X-Frame-Options` header is set by Mattermost itself and cannot be disabled via Mattermost configuration.</Warning>

***

## TCP Proxy Management

The Mattermost PostgreSQL database has a TCP proxy for emergency direct access.

| Property | Value                       |
| -------- | --------------------------- |
| Host     | `shinkansen.proxy.rlwy.net` |
| Port     | `27531`                     |
| User     | `mattermost`                |
| Database | `mattermost`                |
| SSL      | Disabled                    |

### When to Enable

* Password resets for locked-out admin accounts
* Direct role modifications not possible via the API
* Database migration troubleshooting
* One-time data corrections

### When to Disable

Disable the TCP proxy whenever it is not actively needed for admin operations.

### How to Toggle

1. Open the **equa-mattermost** Railway project
2. Navigate to the **PostgreSQL** service
3. Go to **Settings > Networking > Public Networking**
4. Toggle the TCP proxy off (disable) or on (enable)

### Risk Assessment

| State    | Risk                                    | Mitigation                                                                                             |
| -------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| Enabled  | Database reachable from public internet | Protected by database credentials; but credentials could be leaked via env vars, logs, or config drift |
| Disabled | No remote DB access for emergency ops   | Use Railway built-in query interface or re-enable temporarily                                          |

**Recommendation:** Keep disabled by default. Enable only for specific admin tasks, then disable immediately after.

***

## Scaling Considerations

Mattermost Team Edition on Railway Hobby plan:

| Resource    | Current                  | Limit                        | Action if exceeded                               |
| ----------- | ------------------------ | ---------------------------- | ------------------------------------------------ |
| RAM         | \~512MB                  | Hobby plan limits            | Upgrade Railway plan or migrate to dedicated VPS |
| Storage     | Volume-backed            | Hobby plan limits            | Monitor via Railway dashboard                    |
| Connections | Low (single org testing) | PostgreSQL connection limits | Add connection pooling if needed                 |

For production scale (50+ concurrent users), consider:

* Upgrading to Railway Pro plan
* Adding Redis for session caching (`MM_CACHEETTINGS_CACHETYPE=redis`)
* Configuring S3 for file storage (`MM_FILESETTINGS_DRIVERNAME=amazons3`)
* Setting up log aggregation