Repair Temporary File & Cleanup Failures

I have analyzed the lifecycle failure in your background worker. The primary issue is the lack of a formal cleanup contract and the absence of atomic publication, which leads to orphaned files when the process receives a SIGTERM or crashes during a write operation.

Temporary File Lifecycle Repair

The following implementation uses a context manager to ensure cleanup and an atomic write-then-rename pattern to prevent partial data corruption.

import os
import tempfile
import logging
from contextlib import contextmanager
from pathlib import Path

logger = logging.getLogger(__name__)

@contextmanager
def atomic_write_path(target_path, dir=None):
    """
    Ensures a file is written completely or not at all.
    Handles cleanup of the temporary file even on exceptions or crashes.
    """
    target_path = Path(target_path)
    # Create temp file in the same filesystem as target for atomic rename
    fd, temp_path = tempfile.mkstemp(
        dir=dir or target_path.parent, 
        prefix=f".{target_path.name}-",
        suffix=".tmp"
    )
    try:
        os.close(fd) # Close handle so we can use standard file IO
        yield Path(temp_path)
        
        # Ensure data is flushed to disk before renaming
        with open(temp_path, 'rb+') as f:
            f.flush()
            os.fsync(f.fileno())
            
        os.replace(temp_path, target_path)
    except Exception as e:
        logger.error(f"Failed to write {target_path}: {e}")
        raise
    finally:
        # Final safety cleanup if rename didn't happen
        if os.path.exists(temp_path):
            try:
                os.unlink(temp_path)
            except OSError:
                pass

# Usage in your worker
def process_data(data, output_file):
    with atomic_write_path(output_file) as tmp:
        with open(tmp, 'w') as f:
            f.write(data)

Next steps

1. Integrate the atomic_write_path utility into your main processing loop.
2. Register a signal handler for SIGTERM to allow the context manager's finally block to execute.
3. Verify that your /tmp directory and the target directory reside on the same mount point to ensure os.replace is atomic.