Dual Writes & Data Synchronization

Dual writes are a critical technique for migrating data between systems while keeping both systems synchronized. When you can't migrate everything at once—whether due to risk, complexity, or deployment constraints—dual writes allow you to write data to both old and new systems simultaneously, ensuring consistency during the transition period.

This pattern is essential for database migrations, service migrations, and any scenario where you need to keep multiple data stores in sync while traffic gradually shifts from one to the other.

What Are Dual Writes?

The Concept

Dual write: Every write operation updates both the old and new system.

async function saveUser(user: User): Promise<void> {
  // Write to BOTH systems
  await Promise.all([
    legacyDatabase.saveUser(user),
    modernDatabase.saveUser(user),
  ]);
}

Why this helps:

Both systems stay synchronized
Can read from either system during migration
Can switch read traffic gradually
Easy to roll back if problems arise
Validates new system with production data

Dual Writes vs. Other Approaches

Dual Writes:

Application writes to both systems
Fast, immediate consistency
Couples application to both systems temporarily
Simple to implement

Change Data Capture (CDC):

Database-level replication
Application doesn't know about sync
May have replication lag
More complex infrastructure

Event Sourcing:

Write events, multiple systems consume
Eventual consistency
More flexible but more complex
Good for event-driven architectures

Implementation Patterns

Pattern 1: Simple Dual Write

class UserRepository {
  constructor(
    private legacyDb: LegacyDatabase,
    private modernDb: ModernDatabase
  ) {}

  async createUser(user: User): Promise<void> {
    // Write to both
    await Promise.all([
      this.legacyDb.insert('users', user),
      this.modernDb.insert('users', user),
    ]);
  }

  async updateUser(userId: string, updates: Partial<User>): Promise<void> {
    // Write to both
    await Promise.all([
      this.legacyDb.update('users', userId, updates),
      this.modernDb.update('users', userId, updates),
    ]);
  }

  async deleteUser(userId: string): Promise<void> {
    // Delete from both
    await Promise.all([
      this.legacyDb.delete('users', userId),
      this.modernDb.delete('users', userId),
    ]);
  }
}

Pattern 2: Primary/Secondary with Error Handling

class UserRepository {
  async createUser(user: User): Promise<void> {
    // Primary write (legacy) - must succeed
    await this.legacyDb.insert('users', user);

    // Secondary write (modern) - log errors but don't fail request
    try {
      await this.modernDb.insert('users', user);
      metrics.increment('dual_write.success');
    } catch (error) {
      logger.error('Modern DB write failed', { userId: user.id, error });
      metrics.increment('dual_write.failure');

      // Queue for retry
      await this.retryQueue.enqueue({
        operation: 'createUser',
        data: user,
      });
    }
  }
}

Pattern 3: Feature Flag Controlled

class UserRepository {
  async createUser(user: User): Promise<void> {
    // Always write to legacy
    await this.legacyDb.insert('users', user);

    // Conditionally write to modern
    if (featureFlags.isEnabled('dual-write-to-modern-db')) {
      try {
        await this.modernDb.insert('users', user);
      } catch (error) {
        logger.error('Modern DB write failed', { error });
        // Don't fail the request
      }
    }
  }

  async getUser(userId: string): Promise<User> {
    // Read from primary source based on flag
    if (featureFlags.isEnabled('read-from-modern-db')) {
      return this.modernDb.findById(userId);
    }
    return this.legacyDb.findById(userId);
  }
}

Pattern 4: Transactional Dual Write

class OrderRepository {
  async createOrder(order: Order): Promise<void> {
    // Use distributed transaction if possible
    const transaction = await this.transactionManager.begin();

    try {
      await transaction.execute(
        this.legacyDb,
        'INSERT INTO orders VALUES (?, ?, ?)',
        [order.id, order.customerId, order.total]
      );

      await transaction.execute(
        this.modernDb,
        'INSERT INTO orders VALUES (?, ?, ?)',
        [order.id, order.customerId, order.total]
      );

      await transaction.commit();
    } catch (error) {
      await transaction.rollback();
      throw error;
    }
  }
}

Data Consistency Strategies

Strategy 1: Reconciliation Jobs

Periodically verify and fix inconsistencies:

class DataReconciliationJob {
  async run() {
    const batchSize = 1000;
    let offset = 0;

    while (true) {
      const legacyUsers = await this.legacyDb.query(
        'SELECT * FROM users ORDER BY id LIMIT ? OFFSET ?',
        [batchSize, offset]
      );

      if (legacyUsers.length === 0) break;

      for (const legacyUser of legacyUsers) {
        const modernUser = await this.modernDb.findById(legacyUser.id);

        if (!modernUser) {
          // Missing in modern DB, copy over
          logger.warn('User missing in modern DB', { userId: legacyUser.id });
          await this.modernDb.insert('users', legacyUser);
          metrics.increment('reconciliation.missing_record');
        } else if (!this.areEqual(legacyUser, modernUser)) {
          // Data mismatch, use legacy as source of truth
          logger.warn('User data mismatch', {
            userId: legacyUser.id,
            legacy: legacyUser,
            modern: modernUser,
          });
          await this.modernDb.update('users', legacyUser.id, legacyUser);
          metrics.increment('reconciliation.data_mismatch');
        }
      }

      offset += batchSize;
    }

    logger.info('Reconciliation complete', { recordsChecked: offset });
  }

  private areEqual(a: any, b: any): boolean {
    // Implement deep comparison logic
    return JSON.stringify(a) === JSON.stringify(b);
  }
}

Strategy 2: Checksums and Validation

class UserRepository {
  async createUser(user: User): Promise<void> {
    const checksum = this.calculateChecksum(user);

    await Promise.all([
      this.legacyDb.insert('users', { ...user, checksum }),
      this.modernDb.insert('users', { ...user, checksum }),
    ]);

    // Async validation
    this.validateDualWrite(user.id, checksum);
  }

  private async validateDualWrite(userId: string, expectedChecksum: string) {
    // Wait a bit for replication
    await sleep(100);

    const [legacyUser, modernUser] = await Promise.all([
      this.legacyDb.findById(userId),
      this.modernDb.findById(userId),
    ]);

    if (legacyUser.checksum !== expectedChecksum) {
      logger.error('Legacy write verification failed', { userId });
    }

    if (modernUser.checksum !== expectedChecksum) {
      logger.error('Modern write verification failed', { userId });
    }
  }

  private calculateChecksum(data: any): string {
    return crypto
      .createHash('sha256')
      .update(JSON.stringify(data))
      .digest('hex');
  }
}

Strategy 3: Write-Ahead Log

class WriteAheadLogger {
  async logWrite(operation: WriteOperation): Promise<void> {
    // Log to write-ahead log first
    await this.wal.append({
      id: operation.id,
      timestamp: Date.now(),
      operation: operation.type,
      data: operation.data,
      status: 'pending',
    });

    try {
      // Execute dual write
      await Promise.all([
        this.legacyDb.execute(operation),
        this.modernDb.execute(operation),
      ]);

      // Mark as completed
      await this.wal.update(operation.id, { status: 'completed' });
    } catch (error) {
      // Mark as failed
      await this.wal.update(operation.id, {
        status: 'failed',
        error: error.message,
      });
      throw error;
    }
  }

  // Background job to retry failed operations
  async retryFailedOperations() {
    const failed = await this.wal.findByStatus('failed');

    for (const operation of failed) {
      try {
        await this.logWrite(operation);
      } catch (error) {
        logger.error('Retry failed', { operation, error });
      }
    }
  }
}

Common Pitfalls

1. Performance Impact

Problem: Dual writes double the latency and load.

Solution: Make secondary writes async when possible:

async function saveUser(user: User): Promise<void> {
  // Primary write (blocks response)
  await legacyDb.saveUser(user);

  // Secondary write (async, don't await)
  modernDb
    .saveUser(user)
    .catch((error) => logger.error('Async dual write failed', { error }));
}

2. Partial Failures

Problem: Write succeeds in one system but fails in another.

Solution: Implement retry logic and monitoring:

async function saveUser(user: User): Promise<void> {
  const results = await Promise.allSettled([
    legacyDb.saveUser(user),
    modernDb.saveUser(user),
  ]);

  if (results[0].status === 'rejected') {
    logger.error('Legacy write failed', { userId: user.id });
    throw results[0].reason;
  }

  if (results[1].status === 'rejected') {
    logger.error('Modern write failed - queuing retry', { userId: user.id });
    await retryQueue.enqueue({ operation: 'saveUser', data: user });
  }
}

3. Data Transformation Differences

Problem: Same data needs different formats in different systems.

Solution: Transform data appropriately for each system:

async function saveOrder(order: Order): Promise<void> {
  // Legacy format: denormalized
  const legacyFormat = {
    order_id: order.id,
    customer_name: order.customer.name,
    customer_email: order.customer.email,
    total_amount: order.total,
  };

  // Modern format: normalized
  const modernFormat = {
    id: order.id,
    customerId: order.customer.id,
    total: order.total,
  };

  await Promise.all([
    legacyDb.insert('orders', legacyFormat),
    modernDb.insert('orders', modernFormat),
  ]);
}

4. Ordering Issues

Problem: Operations arrive out of order in secondary system.

Solution: Include sequence numbers or timestamps:

let sequenceNumber = 0;

async function saveUser(user: User): Promise<void> {
  const seq = ++sequenceNumber;

  await Promise.all([
    legacyDb.saveUser({ ...user, seq }),
    modernDb.saveUser({ ...user, seq }),
  ]);
}

// In reconciliation, use sequence to determine latest

Migration Lifecycle

Phase 1: Enable Dual Writes

// Start writing to both systems
const repo = new UserRepository(legacyDb, modernDb);

Phase 2: Backfill Historical Data

async function backfillUsers() {
  const users = await legacyDb.findAll('users');

  for (const user of users) {
    const exists = await modernDb.exists('users', user.id);
    if (!exists) {
      await modernDb.insert('users', user);
    }
  }
}

Phase 3: Validate Consistency

async function validateConsistency() {
  const sample = await legacyDb.sample('users', 1000);

  for (const legacyUser of sample) {
    const modernUser = await modernDb.findById(legacyUser.id);

    assert.deepEqual(legacyUser, modernUser);
  }
}

Phase 4: Switch Read Traffic

// Gradually shift reads to modern DB
function getUser(userId: string): Promise<User> {
  if (featureFlags.isEnabled('read-from-modern-db')) {
    return modernDb.findById(userId);
  }
  return legacyDb.findById(userId);
}

Phase 5: Stop Writing to Legacy

// Only write to modern DB
async function saveUser(user: User): Promise<void> {
  await modernDb.saveUser(user);
  // No longer writing to legacy
}

Phase 6: Decommission Legacy

// Remove legacy DB entirely
class UserRepository {
  constructor(private db: ModernDatabase) {}

  async saveUser(user: User): Promise<void> {
    await this.db.saveUser(user);
  }
}

Key Takeaways

Dual writes keep systems synchronized during migrations
Write to both systems, read from primary until ready to switch
Handle partial failures gracefully with retry logic
Validate consistency with reconciliation jobs
Monitor dual write success rates and performance impact
Use feature flags to control read/write routing
Plan clear phases for enabling, validating, and disabling dual writes
Transform data appropriately for each system's format
Don't maintain dual writes indefinitely—set migration deadlines