Three jobs that come up again

When a connection fails

Database logins from Aker are short-lived, so a refused connection or a rejected password is expected now and then. Read a value when you need it, keep it briefly, and try once more with a fresh one.

const API = 'https://app.aker.tech/api'
const stashId = '…'
const credential = credentialHeaders()   // the header from Authentication

const CACHE_MS = 60_000
let cached: { dsn: string; at: number } | null = null

async function read(name: string): Promise<string> {
  const found = await fetch(`${API}/stashes/${stashId}/secrets/lookup`, {
    method: 'POST',
    headers: { ...credential, 'Content-Type': 'application/json' },
    body: JSON.stringify({ name }),
  }).then((res) => (res.ok ? res.json() : null))
  if (!found) throw new Error(`no secret named ${name}`)

  const hit = await fetch(`${API}/stashes/${stashId}/secrets/${found.id}`, {
    headers: credential,
  })
  if (!hit.ok) throw new Error('refused or expired')
  return (await hit.json()).value
}

async function dsn(force = false): Promise<string> {
  if (!force && cached && Date.now() - cached.at < CACHE_MS) {
    return cached.dsn
  }
  const value = await read('prod-database')
  cached = { dsn: value, at: Date.now() }
  return value
}

async function query(sql: string) {
  try {
    return await run(sql, await dsn())
  } catch (err) {
    if (!isConnectionError(err)) throw err      // ECONNREFUSED, 28P01, ER_ACCESS_DENIED…
    return await run(sql, await dsn(true))      // fresh login, one retry
  }
}

Things worth knowing:

• Every read creates a new login and records it. Read once, then reuse the value until it ages out.
• Set ttlMinutes shorter than your deploy cycle. A leaked connection string then stops working on its own. Without a TTL, the login stays until someone revokes it.
• A second read leaves the first value active. If you still hold it, revoke it: GET …/grants, then POST …/grants/:grantId/revoke. Grants name the person and channel, so the cleanup shows up in the audit trail.
• Running many workers? Wrap the integration in redis cache so they share one value and one expiry.

# shell: retry once with a fresh credential
psql "$(aker-dsn)" -c 'select 1' || psql "$(aker-dsn --refresh)" -c 'select 1'

When a secret leaks

Revoke what was handed out, replace what the secret holds, then fix it at the source. What applies depends on the integration.

1. See what is out there. GET …/secrets/:secretId/grants lists every value issued from the secret, with the person, channel, version, and status. Active rows are credentials that still work.
2. Revoke them. POST …/secrets/:secretId/grants/:grantId/revoke removes the login, user, or cache key that was issued. The leaked value stops working, and the next read creates a new one.
3. Rotate the stored value. POST …/secrets/:secretId/rotate runs the integration's rotate step, saves any new state under the same key, bumps the version, records who did it, and marks earlier values as superseded. No built-in integration implements rotate yet, so today this answers 409. For those secrets, step 2 and step 4 are the rotation.
4. Change a stored value at the source. Replace the key or password where it is issued, then PATCH …/secrets/:secretId with the full rule list and the new value. Editing does not bump the version and clears any state the old rules created.
5. Re-encrypt if storage itself was exposed. POST …/secrets/:secretId/rotate-key gives one secret a new encryption key. It is not available from the app; an administrator rotates encryption keys.
6. Add a limit. Put an expiresAt on the secret so it stops working even if nobody remembers, and change the rules so the same kind of caller cannot get the value again.

Expiry never blocks a rotation. Integrations are given the secret's expiresAt and decide for themselves.

Decoys and honeypots

A decoy works when a caller cannot tell it from the real thing, and when the read is still recorded. Order of rules, generated values, and the audit trail are what you need.

Give suspicious callers a fake value

Put the decoy rule first; a matching caller never reaches the rule below it. Generated values cost nothing to store and differ on every read, so a leaked decoy cannot be matched to a later request.

{ "rules": [
  {
    "id": "decoy-bots",
    "name": "Bots get plausible garbage",
    "match": { "userType": ["bot"], "channel": ["scanner", "crawler"] },
    "action": { "actionset": "generator", "action": "token",
      "params": { "length": 40, "encoding": "alphanumeric", "prefix": "sk_live_" } }
  },
  {
    "id": "decoy-ex-employees",
    "match": { "emails": ["*@old-contractor.example"] },
    "action": { "actionset": "generator", "action": "faker",
      "params": { "module": "internet", "method": "username" } }
  },
  {
    "id": "real",
    "match": { "roles": ["admin", "developer"] },
    "action": { "actionset": "pgsql", "action": "create-role", "params": { … } }
  }
] }

Make the decoy fail the way a real value would fail if it were wrong: a login that is refused, an expired token, a URL that returns 404. Handing back something that works teaches the caller more than it hides.

Leave a canary

Create a secret no code should ever read. Give it an obviously fake value and watch for reads:

{
  "name": "aws-prod-rotate-me",
  "rules": [{ "id": "canary", "match": {},
    "action": { "actionset": "generator", "action": "faker",
      "params": { "module": "internet", "method": "username", "seed": 7 } } }]
}

Any read is a signal. The issued-value list and the audit trail record the person, channel, IP address, and user agent Aker recorded. Alert on any read of this secret, and revoke it when one appears.

Refuse by default, and answer the same way

• A caller with no matching rule gets the same answer as an unknown stash or a secret they cannot see, so a response cannot be used to learn what exists.
• Bad request parameters are refused, never rounded up, so a malformed request cannot ask for more than the rule allows.
• Do not count on response times. Aker does not promise that a read takes the same time whether it succeeded or not, and a decoy is there to make the answer indistinguishable in content.
• Channel and context come from the caller. Use them to route to a decoy and to label the audit trail, never as the only protection on a real value.