Moving Sixteen Hardcoded Pages Into Database Rows Without Losing One

For a while this blog ran on two systems at once. Sixteen posts were hardcoded Django templates, each with its own view function and URL entry, listed on the index from a generated file called blog_data.py. Newer posts were database rows. Both were "the blog".

The symptom

The symptom was not an exception in a log. A one-word typo fix in a legacy post required editing a template, committing and deploying. Adding a cover image to one was impossible: the index rendered legacy posts from blog_data.py, and that file had no image field at all.

That is the real cost of content living in code: not slow, not insecure, simply not editable by the only mechanism the site has for editing content.

So all sixteen went into rows, the view functions and URL entries were deleted, blog_data.py was removed, and the old templates were archived. What follows is the method, because "content living in code needs to become data" is a job we have done on enough Django and Python projects to have a checklist for it.

Inventory from the URL map, not the filesystem

The instinct is to list the template directory and count files. Do not. The filesystem tells you which templates exist, not which URLs the public can reach, and those are two different sets. One of the sixteen had a template filename that did not match the URL segment it was served at; had we generated slugs from filenames - the obvious one-liner - that post would have moved and the old URL would have started returning 404 to every inbound link.

The authoritative list is the URL configuration. Walk it, resolve every pattern pointing at a view function you are about to delete, and record the path each one actually answers. That path becomes the slug, so one generic detail view answers exactly the addresses the sixteen per-post views did.

Capture a BEFORE snapshot

You cannot prove you lost nothing unless you wrote down what was there. Before touching a line we fetch every public URL, extract the visible text as a list of blocks, and store it as JSON. Django's test client goes through the real resolver, views and templates, in-process, with no server to start. One gotcha: it sends Host: testserver, which a production ALLOWED_HOSTS rejects with a 400 before your view runs. Every URL comes back 400 and the snapshot looks fine because it is uniformly empty. Send a host the settings allow.

#!/usr/bin/env python
"""Snapshot every public URL's text, and diff two snapshots.

    python snapshot.py capture before.json /blog/ /blog/some-post/
    python snapshot.py diff before.json after.json
"""
import json
import os
import re
import sys

BLOCK_RE = re.compile(
    r'<(p|h2|h3|li|blockquote|pre)\b[^>]*>(.*?)</\1>', re.S | re.I)
TAG_RE = re.compile(r'<[^>]+>')
WS_RE = re.compile(r'\s+')


def blocks(html):
    """Every visible text block, tags stripped and whitespace normalised."""
    out = []
    for _, inner in BLOCK_RE.findall(html):
        text = WS_RE.sub(' ', TAG_RE.sub(' ', inner)).strip()
        if text:
            out.append(text)
    return out


def capture(urls):
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'agicad.settings')
    import django
    django.setup()
    from django.test import Client

    # The test client sends Host: testserver, which a production
    # ALLOWED_HOSTS will reject with a 400. Send a host the settings allow.
    host = os.environ.get('SNAPSHOT_HOST', 'localhost')
    client = Client()
    snap = {}
    for url in urls:
        response = client.get(url, follow=True, HTTP_HOST=host)
        body = response.content.decode('utf-8', 'replace')
        snap[url] = {'status': response.status_code, 'blocks': blocks(body)}
        print('%s %s %d blocks' % (response.status_code, url, len(snap[url]['blocks'])))
    return snap


def diff(before, after):
    problems = []
    for url, old in before.items():
        new = after.get(url)
        if new is None:
            problems.append('%s: missing from the after snapshot' % url)
            continue
        if new['status'] != old['status']:
            problems.append('%s: status %s -> %s' % (url, old['status'], new['status']))
        lost = [b for b in old['blocks'] if b not in new['blocks']]
        for b in lost:
            problems.append('%s: lost block %r' % (url, b[:80]))
    return problems


def load(path):
    with open(path, encoding='utf-8') as fh:
        return json.load(fh)


if __name__ == '__main__':
    mode, target = sys.argv[1], sys.argv[2]
    if mode == 'capture':
        with open(target, 'w', encoding='utf-8') as fh:
            json.dump(capture(sys.argv[3:]), fh, indent=1, sort_keys=True)
        sys.exit(0)
    before = load(target)
    found = diff(before, load(sys.argv[3]))
    for problem in found:
        print('FAIL', problem)
    print('%d URL(s) compared, %d problem(s)' % (len(before), len(found)))
    sys.exit(1 if found else 0)

Comparing block lists rather than raw HTML matters. Raw HTML changes for innocent reasons - a wrapper div, whitespace, a CSRF token. Extracted text changes when you lose a paragraph.

Write the importer as an idempotent command

Make it a management command, not a data migration and not a script you throw away. It has to survive running twice, on a fresh database, after a rollback and in CI - which means a stable business key. The slug is that key; an auto-increment id is not, because ids differ between laptop, staging and production.

The clobber question

Every importer has to answer one question, and most answer it by accident: once a human can also edit the row, who wins?

Our answer is a fingerprint. On every write the importer stores a SHA-256 digest of the field values it wrote. On the next run it re-digests the row's current values. A match means nobody has touched the row, so the file wins. A mismatch means a human edited it, so the importer skips it with a warning rather than destroying somebody's work. --force exists for when you mean it.

Two details make the digest trustworthy: the field names go into the hash alongside the values, so moving a value between fields changes it; and the field order is declared once and fixed.

"""Import content/articles/*.html into Article rows, idempotently."""
import hashlib
import os
from datetime import date

from django.conf import settings
from django.core.management.base import BaseCommand

from website.models import Article

#: The fields the importer owns. The ORDER is part of the fingerprint.
MANAGED_FIELDS = ('title', 'category', 'body_html', 'published_at')


class ArticleError(ValueError):
    """The source file is not usable."""


def parse_article(text, filename='<string>'):
    """(front matter dict, body html)."""
    stripped = text.lstrip(' \t\r\n')
    if not stripped.startswith('<!--'):
        raise ArticleError('%s: file must start with an <!-- ... --> header' % filename)
    end = stripped.find('-->')
    if end == -1:
        raise ArticleError('%s: header is never closed with -->' % filename)

    header, body = stripped[4:end], stripped[end + 3:].strip()
    meta = {'category': 'Engineering'}
    for lineno, raw in enumerate(header.splitlines(), start=2):
        line = raw.strip()
        if not line:
            continue
        if ':' not in line:
            raise ArticleError('%s line %d: expected "key: value"' % (filename, lineno))
        key, value = line.split(':', 1)
        meta[key.strip().lower()] = value.strip()

    for key in ('title', 'slug', 'publish_date'):
        if not meta.get(key):
            raise ArticleError('%s: missing required header key %r' % (filename, key))
    if not body:
        raise ArticleError('%s: body is empty' % filename)

    try:
        y, m, d = (int(part) for part in meta['publish_date'].split('-'))
        meta['publish_date'] = date(y, m, d)
    except (TypeError, ValueError):
        raise ArticleError('%s: publish_date must be YYYY-MM-DD' % filename)
    return meta, body


def article_to_fields(meta, body):
    return {
        'title': meta['title'],
        'category': meta['category'],
        'body_html': body,
        'published_at': meta['publish_date'],
    }


def fingerprint(values):
    """Stable digest of the managed field values. Names are hashed too."""
    h = hashlib.sha256()
    for name in MANAGED_FIELDS:
        h.update(name.encode('utf-8'))
        h.update(b'\x00')
        h.update(str(values[name]).encode('utf-8'))
        h.update(b'\x1e')
    return h.hexdigest()


def row_fingerprint(row):
    return fingerprint({name: getattr(row, name) for name in MANAGED_FIELDS})


class Command(BaseCommand):
    help = 'Upsert content/articles/*.html into Article rows (idempotent).'

    def add_arguments(self, parser):
        parser.add_argument('--dry-run', action='store_true')
        parser.add_argument('--force', action='store_true')

    def handle(self, *args, **options):
        directory = os.path.join(settings.BASE_DIR, 'content', 'articles')
        dry_run, force = options['dry_run'], options['force']
        created = updated = unchanged = skipped = failed = 0

        if not os.path.isdir(directory):
            self.stderr.write('no such directory: %s' % directory)
            return

        seen = {}
        for name in sorted(os.listdir(directory)):
            if not name.endswith('.html'):
                continue
            path = os.path.join(directory, name)
            with open(path, encoding='utf-8') as fh:
                text = fh.read()
            try:
                meta, body = parse_article(text, name)
            except ArticleError as exc:
                failed += 1
                self.stderr.write('ERROR %s' % exc)
                continue

            slug = meta['slug']
            if slug in seen:
                failed += 1
                self.stderr.write('ERROR %s: slug %r is %s\'s'
                                  % (name, slug, seen[slug]))
                continue
            seen[slug] = name

            fields = article_to_fields(meta, body)
            wanted = fingerprint(fields)
            row = Article.objects.filter(slug=slug).first()

            if row is None:
                created += 1
                self.stdout.write('create %s' % slug)
                if not dry_run:
                    Article.objects.create(slug=slug, source_file=name,
                                           fingerprint=wanted, **fields)
                continue

            if not row.source_file:
                skipped += 1
                self.stderr.write('WARNING %s: slug %r was authored in the admin'
                                  % (name, slug))
                continue

            current = row_fingerprint(row)
            if current == wanted and row.fingerprint == wanted:
                unchanged += 1
                continue
            if row.fingerprint and current != row.fingerprint and not force:
                skipped += 1
                self.stderr.write('WARNING %s: row %r edited by hand since the last '
                                  'import; skipping (--force overwrites)'
                                  % (name, slug))
                continue

            updated += 1
            self.stdout.write('update %s' % slug)
            if not dry_run:
                for key, value in fields.items():
                    setattr(row, key, value)
                row.source_file = name
                row.fingerprint = wanted
                row.save()

        self.stdout.write('%screated=%d updated=%d unchanged=%d skipped=%d errors=%d'
                          % ('[dry-run] ' if dry_run else '',
                             created, updated, unchanged, skipped, failed))

Note the third case in handle(): a row with an empty source_file was authored in the admin and is never touched. An importer that owns rows it did not create will eventually eat something. We ran the digest logic on its own before trusting it with content:

digest       0dd2dca46699ab68df95961f32bfbf5b6f368fd1edcbaf4913c8476816039d2a
reordered    23e32955dacf2530258c1fdea7c2abd6c21acf2eaab3a696a2daec881904834b
order counts True
round-trip   True
human edit   False <- False: edit detected
swapped      True <- names are hashed

Prove parity, then delete the old path

After the import, snapshot again and diff. On the real migration that meant fetching all eighteen post URLs before and after: zero lost blocks, every URL still a 200. Here is the harness, run against a clean snapshot and then a deliberately damaged one so we know the diff fails when it should:

$ PYTHONPATH=. venv/bin/python snapshot.py capture before.json \
      /about/ /services/ /contact/ /web-app-development/
200 /about/ 87 blocks
200 /services/ 110 blocks
200 /contact/ 40 blocks
200 /web-app-development/ 144 blocks

$ venv/bin/python snapshot.py diff before.json after.json
4 URL(s) compared, 0 problem(s)

$ venv/bin/python snapshot.py diff before.json after-with-a-deliberate-regression.json
FAIL /contact/: lost block 'case studies'
FAIL /services/: status 200 -> 404
4 URL(s) compared, 2 problem(s)

Then, in the same change and not in a follow-up ticket, delete the old path: view functions, URL entries, generated index data. Leaving both alive because "we can clean it up later" is how you get two sources of truth again. Archive the old templates rather than deleting them - they are the only record of what a page used to look like if the diff ever disagrees with you.

One bug the migration fixed for free

Post bodies are stored as raw HTML and rendered with |safe, never passed through the template engine. That fixed a live bug: one post contained GitHub Actions examples, and while it was a template the engine ate ${{ secrets.PROD_HOST }} as a variable expression and rendered the line as host: $. Content is not a template - the moment an author can write braces, running their text through a template engine is a rendering bug waiting to happen.

The consequence we accepted is worth stating plainly: nothing in the repo validates the HTML in that column. The admin is the only editor and it is superuser-only, so the trust boundary is "anyone who can log in as a superuser can already do worse". With untrusted authors that answer is wrong and you want sanitisation on save.

When not to do this

If the content never changes and has nobody to edit it - a legal notice, a single landing page - a template is fine. Better than fine: it is version-controlled and diffable, which a database row is not. A migration you do not need is pure risk with no payoff.

The trigger is not "this content is in a template". It is "somebody who is not a developer needs to change it, and cannot".

If you have an archive stuck in code and a deploy standing between your client and a typo fix, that is the kind of work we take on across our development services. Start a project.