📧 Broadcast Email Deduplication Guide

# BROADCAST EMAIL DEDUPLICATION GUIDE
# Sådan sikrer du at der KUN sendes 1 email pr. unik modtager
# Sidst opdateret: 2026-04-02

=====================================================================
PROBLEMET
=====================================================================

Broadcast-kampagner afsendes via et cron-job hvert X minut.
Uden beskyttelse kan det samme kald (eller to overlappende kald)
sende den samme email to gange.

Risici:
- [HØJ]    Duplikerede modtagere i BroadcastRecipient tabellen
- [MEDIUM] Race condition: to overlappende cron-kald behandler samme modtager
- [HØJ]    Retry uden statustjek sender emails der allerede er sendt


=====================================================================
LØSNINGEN — 3 BESKYTTELSESLAG
=====================================================================

LAG 1: In-memory deduplication
  → Dedupliker alle recipients i memory FØR afsendelse (email_normalized)

LAG 2: Claim-before-send (UDEN re-fetch)
  → Sæt sent: true FØR Brevo-kald. Revert til sent: false ved fejl.
  → VIGTIGT: Lav IKKE et ekstra .filter({ id }) re-fetch per recipient —
    det koster 1 ekstra DB API-kald per email og rammer Base44's rate limit (429).
    In-memory deduplication (LAG 1) er tilstrækkelig beskyttelse mod dobbelt-send.

LAG 3: Rate limit + cron interval
  → 50 emails per kald, 400ms delay, 2 minutters cron interval


=====================================================================
ENTITETS-FLOW
=====================================================================

BroadcastCampaign (status: draft → sending → sent)
    ↓
BroadcastRecipient (will_receive: true, sent: false → sent: true)
    ↓
Brevo API (afsender selve emailen)

BroadcastCampaign felter:
  status: 'draft' | 'sending' | 'sent'
  sent_count: number
  failed_count: number
  will_receive_count: number
  sending_started_at: datetime
  sent_at: datetime

BroadcastRecipient felter:
  campaign_id: string
  email: string
  email_normalized: string   ← KRITISK (lowercase/trimmed)
  will_receive: boolean
  sent: boolean
  failed: boolean
  sent_at: datetime
  error_message: string


=====================================================================
LAG 1: IN-MEMORY DEDUPLICATION (kode)
=====================================================================

// Hent ALLE unsent recipients (pagineret — VIGTIGT!)
const allRecipients = [];
let skip = 0;
while (true) {
  const batch = await base44.asServiceRole.entities.BroadcastRecipient.filter(
    { campaign_id: campaign.id, will_receive: true, sent: false },
    '-created_date', 500, skip
  );
  if (!batch || batch.length === 0) break;
  allRecipients.push(...batch);
  if (batch.length < 500) break;
  skip += 500;
}

// Dedupliker baseret på email_normalized
const seenEmails = new Set();
const recipients = [];
const duplicates = [];

for (const r of allRecipients) {
  const key = (r.email_normalized || r.email || '').toLowerCase().trim();
  if (!key || seenEmails.has(key)) { duplicates.push(r); continue; }
  seenEmails.add(key);
  recipients.push(r);
}

// Marker duplikater som sent
if (duplicates.length > 0) {
  await Promise.all(duplicates.map(r =>
    base44.asServiceRole.entities.BroadcastRecipient.update(r.id, {
      sent: true,
      sent_at: new Date().toISOString(),
      error_message: 'Skipped: duplicate email address'
    })
  ));
}

FALDGRUBER:
! Brug email_normalized (lowercase/trimmed) som nøgle — IKKE rå email
! Husk paginering — stop for tidligt og duplikater overlever på tværs af sider


=====================================================================
LAG 2: CLAIM-BEFORE-SEND (kode) — UDEN re-fetch
=====================================================================

⚠️ KRITISK: Lav IKKE et .filter({ id: recipient.id }) re-fetch per recipient!
   Det koster 1 ekstra Base44 API-kald per email → 50 ekstra kald per run →
   rammer hurtigt Base44's rate limit (429) og stopper hele kampagnen.

for (const recipient of recipientsToSend) {

  // CLAIM — sæt sent: true FØR Brevo-kald (ingen re-fetch)
  await base44.asServiceRole.entities.BroadcastRecipient.update(recipient.id, {
    sent: true,
    sent_at: new Date().toISOString(),
    error_message: 'Sending...'
  });

  try {
    await sendViaBrevo(recipient, campaign);

    // SUCCESS — bekræft og ryd placeholder
    await base44.asServiceRole.entities.BroadcastRecipient.update(recipient.id, {
      sent: true, failed: false, error_message: null
    });

  } catch (error) {
    // FEJL — REVERT claim så næste kald kan retry
    await base44.asServiceRole.entities.BroadcastRecipient.update(recipient.id, {
      sent: false, failed: true, error_message: error.message
    });
  }
}

FALDGRUBER:
! Glemmer du at revertere sent: false ved fejl, hænger recipients i "Sending..." limbo
! Lav ALDRIG .filter({ id }) re-fetch per recipient — forårsager Base44 rate limit (429)


=====================================================================
LAG 3: RATE LIMITING OG CRON INTERVAL
=====================================================================

Konfiguration:
  maxEmailsPerRun = 50
  batchSize       = 10
  delayMsBetween  = 400ms
  cronInterval    = 2 minutter

Beregning:
  50 emails × ~400ms = ~20 sekunder per kald
  Kald tager 20 sek, interval er 120 sek → 100 sek buffer → ingen overlap
  ~150 emails/minut (Brevo grænse: ~300/min)
  ~1.500 emails/time, ~36.000 emails/dag

Rate limit håndtering:
  let rateLimitBackoff = 1;

  if (brevoResponse.status === 429) {
    rateLimitBackoff = Math.min(rateLimitBackoff * 2, 10);
    await new Promise(r => setTimeout(r, 2000 * rateLimitBackoff));
    throw new Error('Rate limit');
  }

  rateLimitBackoff = 1; // nulstil efter success
  await new Promise(r => setTimeout(r, 400 * rateLimitBackoff)); // delay mellem batches

FALDGRUBER:
! Sæt ALDRIG cron til under 1 minuts interval
! Base44 functions timeout efter ~30 sekunder


=====================================================================
FULD KODE TEMPLATE — resumeBroadcastCampaigns
=====================================================================

import { createClientFromRequest } from 'npm:@base44/sdk@0.8.23';

Deno.serve(async (req) => {
  try {
    const base44 = createClientFromRequest(req);
    const BREVO_API_KEY = Deno.env.get('BREVO_API_KEY');

    const campaigns = await base44.asServiceRole.entities.BroadcastCampaign.filter({ status: 'sending' });
    if (campaigns.length === 0) return Response.json({ message: 'No campaigns to process' });

    for (const campaign of campaigns) {

      // 1. Hent alle unsent recipients (pagineret)
      const allRecipients = [];
      let skip = 0;
      while (true) {
        const batch = await base44.asServiceRole.entities.BroadcastRecipient.filter(
          { campaign_id: campaign.id, will_receive: true, sent: false },
          '-created_date', 500, skip
        );
        if (!batch || batch.length === 0) break;
        allRecipients.push(...batch);
        if (batch.length < 500) break;
        skip += 500;
      }

      // 2. In-memory deduplication (LAG 1)
      const seenEmails = new Set();
      const recipients = [], duplicates = [];
      for (const r of allRecipients) {
        const key = (r.email_normalized || r.email || '').toLowerCase().trim();
        if (!key || seenEmails.has(key)) { duplicates.push(r); continue; }
        seenEmails.add(key);
        recipients.push(r);
      }
      if (duplicates.length > 0) {
        await Promise.all(duplicates.map(r =>
          base44.asServiceRole.entities.BroadcastRecipient.update(r.id, {
            sent: true, sent_at: new Date().toISOString(), error_message: 'Skipped: duplicate'
          })
        ));
      }

      // 3. Begræns til max 50 per kald (LAG 3)
      const toSend = recipients.slice(0, 50);
      let sentCount = 0, failedCount = 0, rateLimitBackoff = 1;

      for (let i = 0; i < toSend.length; i += 10) {
        const batch = toSend.slice(i, i + 10);

        for (const recipient of batch) {

          // 4. Claim-before-send (LAG 2) — INGEN re-fetch (undgår Base44 rate limit)
          await base44.asServiceRole.entities.BroadcastRecipient.update(recipient.id, {
            sent: true, sent_at: new Date().toISOString(), error_message: 'Sending...'
          });

          try {
            const htmlContent = campaign.html_content
              .replace(/{{firstName}}/g, recipient.first_name || '')
              .replace(/{{email}}/g, recipient.email || '');

            const brevoRes = await fetch('https://api.brevo.com/v3/smtp/email', {
              method: 'POST',
              headers: { 'Content-Type': 'application/json', 'api-key': BREVO_API_KEY },
              body: JSON.stringify({
                sender: { name: campaign.from_name, email: campaign.from_email },
                to: [{ email: recipient.email }],
                subject: campaign.subject,
                htmlContent
              })
            });

            if (brevoRes.status === 429) {
              rateLimitBackoff = Math.min(rateLimitBackoff * 2, 10);
              await new Promise(r => setTimeout(r, 2000 * rateLimitBackoff));
              throw new Error('Rate limit');
            }
            if (!brevoRes.ok) throw new Error('Brevo: ' + brevoRes.status);

            await base44.asServiceRole.entities.BroadcastRecipient.update(recipient.id, {
              sent: true, failed: false, error_message: null
            });
            sentCount++;
            rateLimitBackoff = 1;

          } catch (error) {
            await base44.asServiceRole.entities.BroadcastRecipient.update(recipient.id, {
              sent: false, failed: true, error_message: error.message
            });
            failedCount++;
          }
        }

        if (i + 10 < toSend.length) {
          await new Promise(r => setTimeout(r, 400 * rateLimitBackoff));
        }
      }

      // 5. Tjek om kampagnen er færdig
      const remaining = await base44.asServiceRole.entities.BroadcastRecipient.filter(
        { campaign_id: campaign.id, will_receive: true, sent: false }, '-created_date', 1
      );
      await base44.asServiceRole.entities.BroadcastCampaign.update(campaign.id, {
        status: remaining.length === 0 ? 'sent' : 'sending',
        sent_count: (campaign.sent_count || 0) + sentCount,
        failed_count: (campaign.failed_count || 0) + failedCount,
        ...(remaining.length === 0 ? { sent_at: new Date().toISOString() } : {})
      });
    }

    return Response.json({ success: true });

  } catch (error) {
    return Response.json({ error: error.message }, { status: 500 });
  }
});


=====================================================================
TJEKLISTE
=====================================================================

DATABASE / ENTITETER:
  [ ] BroadcastRecipient har email_normalized felt (lowercase/trimmed)
  [ ] BroadcastRecipient har sent (boolean), failed (boolean), error_message (string)
  [ ] BroadcastCampaign har status enum: draft, sending, sent
  [ ] BroadcastCampaign har sent_count, failed_count, will_receive_count felter

BACKEND FUNKTION:
  [ ] Pagineret hentning af recipients (while-loop, 500 per side)
  [ ] In-memory deduplication på email_normalized FØR afsendelse
  [ ] Duplikater markeres sent: true med error_message: "Skipped: duplicate"
  [ ] Max 50 recipients per kald
  [ ] Claim-before-send: sæt sent: true med "Sending..." FØR Brevo-kald
  [ ] INGEN re-fetch (.filter({ id })) per recipient — forårsager Base44 rate limit!
  [ ] SUCCESS path: opdater med failed: false, error_message: null
  [ ] FEJL path: revert til sent: false, failed: true, error_message
  [ ] 429 rate limit håndtering med eksponentiel backoff
  [ ] 400ms delay mellem batches af 10
  [ ] Kampagne markeres status: sent når alle recipients er behandlet

CRON JOB:
  [ ] Interval: 2 minutter (IKKE under 1 minut)
  [ ] Cron kalder KUN resumeBroadcastCampaigns
  [ ] Ingen andre aktive cron jobs med legacy email-sending

DEPREKEREDE FUNKTIONER (skal deaktiveres):
  [ ] processScheduledEmails — returner 200 no-op
  [ ] sendEmailBatch — returner 410 Gone
  [ ] sendEmailBatchV3 — returner 410 Gone


=====================================================================
FEJLSCENARIER OG LØSNINGER
=====================================================================

SCENARIO 1: Emails sendt dobbelt
  Symptom: Modtagere klager over at have modtaget samme email 2 gange
  Årsag:   Manglende claim-before-send ELLER cron interval for kort
  Løsning: Implementer claim-before-send (uden re-fetch) + sæt cron til 2 minutter

SCENARIO 2: Kampagne hænger på status "sending" / sender ikke
  Symptom: Kampagne viser "sending" men ingen emails sendes, sent_count stiger ikke
  Årsag A: Recipients er i limbo — sent: true med "Sending..." placeholder
  Årsag B: Base44 rate limit (429) — for mange DB-kald per run (fx re-fetch per recipient)
  Løsning A: Kør recovery script:
  Løsning B: Fjern re-fetch (.filter({ id })) per recipient fra koden

  const stuck = await base44.asServiceRole.entities.BroadcastRecipient.filter({
    campaign_id: 'DIN_CAMPAIGN_ID',
    error_message: 'Sending...'
  });
  await Promise.all(stuck.map(r =>
    base44.asServiceRole.entities.BroadcastRecipient.update(r.id, {
      sent: false, failed: false, error_message: null
    })
  ));

SCENARIO 3: Mange fejlede recipients
  Symptom: failed_count stiger hurtigt
  Årsag:   Brevo rate limit (429) eller ugyldige emails
  Løsning: Tjek error_message på recipients. Sænk maxEmailsPerRun til 30.

SCENARIO 4: Duplikater i database
  Symptom: Samme email har to BroadcastRecipient records
  Årsag:   generateBroadcastRecipients er kørt to gange
  Løsning: In-memory deduplication (LAG 1) håndterer dette automatisk.
           De markeres med error_message: "Skipped: duplicate"