Caching Generated Mock Fixtures in CI

Every CI run regenerates the same mock fixtures from scratch, adding a minute or more to a job whose inputs have not changed since the last green build. Worse, the naive fixes — committing the fixtures or caching them on a branch key — either invite silent drift or serve stale data after the API contract moves. The goal is a cache that hits when the contract is unchanged and misses exactly when it changes.

Why fixtures get regenerated needlessly

Generated fixtures are a pure function of two things: the API specification and the generator that reads it. If neither has changed, the output is identical, so recomputing it is wasted work. Three habits cause the waste:

  • No cache at all. The pipeline runs generate:fixtures unconditionally as a build step, so a spec-unchanged run pays full generation cost.
  • A cache keyed on the wrong thing. Keying on the branch name or a lockfile hash means the cache neither invalidates on a spec change (serving stale fixtures) nor hits across branches that share the same spec.
  • Non-deterministic generation. If the generator is unseeded, two runs on the same spec produce different bytes, so even a correct cache key restores fixtures that no longer match what a fresh generation would produce — and snapshot tests flap.

The fix is a cache keyed on a hash of the schema-driven generation inputs, combined with a seeded generator so the cached output is reproducible. The generation itself relies on deterministic seed management; without a fixed seed the cache is meaningless.

Solution

1. Make generation deterministic

Pin the seed so identical inputs yield byte-identical fixtures. Anything less and the cache cannot be trusted:

// scripts/generate-fixtures.ts
import { faker } from '@faker-js/faker';
import { writeFileSync, mkdirSync } from 'node:fs';

faker.seed(Number(process.env.MOCK_SEED ?? 42));
mkdirSync('mocks/generated', { recursive: true });

const users = Array.from({ length: 50 }, () => ({
  id: faker.string.uuid(),
  name: faker.person.fullName(),
  email: faker.internet.email(),
}));

// Stable key order + trailing newline => reproducible bytes across runs.
writeFileSync(
  'mocks/generated/users.json',
  JSON.stringify(users, null, 2) + '\n',
);
console.log(`Wrote ${users.length} users`);

2. Key the cache on the spec hash

actions/cache restores a directory when the key matches a previously saved entry. Build the key from a hash of the spec and the generator version so it changes precisely when either input does:

# .github/workflows/fixtures.yml
name: fixtures
on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    env:
      MOCK_SEED: "42"
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"
      - run: npm ci

      - name: Restore fixture cache
        id: cache
        uses: actions/cache@v4
        with:
          path: mocks/generated
          key: fixtures-v3-${{ hashFiles('specs/api.yaml', 'scripts/generate-fixtures.ts') }}
          restore-keys: |
            fixtures-v3-

      - name: Generate fixtures (cache miss only)
        if: steps.cache.outputs.cache-hit != 'true'
        run: npx tsx scripts/generate-fixtures.ts

      - name: Validate fixtures against the spec
        run: npx tsx scripts/validate-fixtures.ts

      - name: Run tests
        run: npm run test:integration

Two details make this robust. The v3 prefix is a manual bust — bump it when you change the generation logic in a way hashFiles cannot see. The restore-keys fallback lets a spec change restore the previous fixtures as a warm starting point (useful for large fixture sets built incrementally), while the exact-key miss still forces a regenerate.

3. Validate restored fixtures against the spec

A cache hit means the bytes match a prior run, not that they match the current contract if someone bumped the v3 prefix incorrectly. Guard every run — hit or miss — with a validation step so stale or mis-keyed fixtures fail fast:

// scripts/validate-fixtures.ts
import Ajv from 'ajv';
import { readFileSync } from 'node:fs';

const ajv = new Ajv({ allErrors: true });
const schema = JSON.parse(readFileSync('specs/user.schema.json', 'utf8'));
const validate = ajv.compile(schema);
const users = JSON.parse(readFileSync('mocks/generated/users.json', 'utf8'));

const bad = users.filter((u: unknown) => !validate(u));
if (bad.length > 0) {
  console.error('Fixtures violate the schema:', ajv.errorsText(validate.errors));
  process.exit(1);
}
console.log(`All ${users.length} fixtures valid against the spec`);

This validation gate is the same contract check described in Detecting OpenAPI Contract Drift in CI — here it catches a stale cache rather than a drifting mock.

Verification

Confirm the cache behaves as intended by checking the step outcome in the run log:

gh run view --log | grep -E "Cache (restored|not found)"
# First run on a new spec:  "Cache not found for input keys: fixtures-v3-<hash>"
# Re-run with unchanged spec: "Cache restored from key: fixtures-v3-<hash>"

A green re-run that prints Cache restored and skips the generate step proves the cache is keyed correctly.

Gotchas and edge cases

  • Stale cache after an out-of-band data change. If you tweak generate-fixtures.ts in a way the hash covers, the key changes automatically — but if you change a data source the hash does not cover (an external CSV, a seed constant in another file), the cache silently serves old fixtures. Add every generation input to hashFiles, or bump the v3 prefix by hand when in doubt.
  • Key collisions across specs. A repo with several specs must namespace the key (fixtures-users-${{ hashFiles('specs/users.yaml') }}) so one service’s fixtures never restore into another’s directory. A single flat key across specs is the classic collision that ships the wrong data.
  • Reproducibility breaks without a stable seed and stable key order. Non-deterministic map iteration or an unseeded faker call makes two “identical” runs produce different bytes, so the cache is never actually reused and snapshot tests flap. Sort object keys on write and pin MOCK_SEED, as in step 1.

← Back to Running Mock Servers in CI Pipelines