7 posts tagged with "Hooks"

Your CSV is not your data. It's a promise that needs verification, transformation, and three layers of validation before it earns the right to touch your database. Here's how a production import hook turns a tab-delimited file into trusted journal entries.

The Premise​

Every data import starts with the same fantasy: upload a file, click import, done. The reality is a defensive pipeline where the database write — the part most people think of as "the import" — is the very last step, and often the simplest one. The hard work is everything that comes before it.

This article dissects a production hook that imports general ledger journal entries from an external accounting system's CSV export. The hook is 300+ lines. Fewer than 20 of those lines actually write to the database.

The Source Data Problem​

Before writing a single line of import code, you need to confront an uncomfortable truth: the source data is your responsibility. The external system exported it. Your users downloaded it. Nobody in that chain guarantees it's clean.

In this pipeline, the source is a tab-delimited CSV with columns like Trans #, Date, GL Code, Debit, Credit, Memo, and Class. Each row is a single journal item line. Multiple rows share the same Trans # to form a compound journal entry.

The file could have:

• Rows with no transaction number (separator lines, subtotals)
• GL codes that don't exist in your chart of accounts
• Amounts formatted with US thousands commas (1,250.00)
• Stray quote characters from copy-paste artifacts
• Zero-amount lines (e.g., Sales Tax Payable 0.00)

The hook's job isn't to "import a CSV." It's to distill a messy file into trusted financial documents — or reject the entire batch with a clear explanation of why.

The Pipeline​

Here's the shape of the import, with the write at the end where it belongs:

Nine steps. Only one writes to the database. Let's walk through them.

Step 1: Validate the Request​

Before downloading or parsing anything, validate that the request itself is well-formed:

bob.validator.joi().validateData(
  { fileUrl: bob.dataPayload.fileUrl },
  SuperJoi.object({
    fileUrl: SuperJoi.string().required()
  }),
  bob.flowUser.getUserInfo()
)

If there's no file URL, fail immediately. Don't download, don't parse, don't allocate memory. Guard clauses first.

Step 2: Load Reference Data​

Before you can validate the CSV, you need to know what "valid" means. That means loading the reference datasets the import will validate against:

// -->Get: all active general ledger accounts
const generalLedgerAccounts = await generalLedgerFlowCollection.docs
  .flowQuery()
  .user(bob.flowUser)
  .addPolicy('canRead')
  .cache()
  .flowOptions(eventOptions.generalLedgerAccountNaoQueryOptions)
  .limit(10000)
  .getMany(undefined, 'FinanceInterface.GeneralLedgerAccount')

// -->Get: warehouses
const warehouses = await warehouseFlowCollection.docs
  .flowQuery()
  .user(bob.flowUser)
  .addPolicy('canRead')
  .getDocumentStatuses(['active'], 'data')
  .cache()
  .flowOptions(eventOptions.warehouseNaoQueryOptions)
  .limit(100)
  .getMany(undefined, 'InventoryManagementInterface.Warehouse')

Notice the .cache() — these datasets are stable reference data. And notice the hard limit. You never query reference data without a ceiling.

Both queries fail fast with descriptive errors if they return empty:

if (generalLedgerAccounts.data.length === 0) {
  throw naoFormatErrorById('bad_request', {
    reason: 'Failed to find general ledger accounts'
  })
}

No GL accounts means no chart of accounts means no point continuing. The error is specific enough that the user knows exactly what to fix.

Step 3: Ensure Preconditions​

Some imports depend on system state that might not exist yet. This hook ensures fiscal periods and years are created before any journal entries are written:

await bob.executeRequestOrThrow(
  eventOptions.ensureFiscalPeriodsActionCfpQuery, {}
)

This is easy to overlook. If you write a journal entry dated March 2024 and no fiscal period exists for Q1 2024, the entry either fails silently or lands in an undefined period. The hook handles this before it even looks at the CSV.

Step 4: Build the Deduplication Index​

For idempotent imports — where re-running the same file shouldn't create duplicates — you need to know what already exists:

const existingJournalEntries = await journalEntryFlowCollection.docs
  .flowQuery()
  .user(bob.flowUser)
  .addPolicy('canRead')
  .flowOptions(eventOptions.journalEntryNaoQueryOptions)
  .query({ 'data.name': { $regex: '^PIPE-' } })
  .projection({ 'data.name': 1 })
  .limit(2_000_000)
  .getMany(undefined, 'FinanceInterface.JournalEntry')

const existingJournalNameSet: Set<string> = new Set(
  existingJournalEntries.data.map((je) => je.data.name)
)

Two things to highlight here. First, the .projection({ 'data.name': 1 }) — we only need the name field, so we only fetch the name field. With 2 million potential records, this is the difference between a 30-second query and a timeout. Second, the Set — O(1) lookup for every deduplication check instead of scanning an array.

Step 5: Parse and Clean​

Now — and only now — do we touch the CSV:

const glCsv = await new NaoCsvRunner2().loadFile({
  inputFilePath: downloadToLocalTemp.filePath,
  delimiter: '\t'
})

const allRowsRaw = await glCsv.readFile({ numberOfLines: Infinity })
const allRows = allRowsRaw.filter(
  (row: any) => cleanValue(row['Trans #'])
)

The cleanValue helper does the unglamorous work that prevents downstream failures:

const cleanValue = (v: string | undefined): string =>
  (v || '').trim().replace(/"+/g, '')

Trim whitespace. Strip stray quotes. Return an empty string instead of undefined. Every field in the CSV passes through this function before it's used anywhere. It's three lines of code that prevent dozens of edge-case bugs.

The filter step drops rows with no Trans # — subtotal rows, blank lines, section headers. These aren't data. They're presentation artifacts from the source system's export.

Step 6: Preflight Validation — The Gate​

This is the most important step in the entire pipeline, and it happens before a single document is created.

The hook scans every row in the file and checks that every GL code exists in the system's chart of accounts:

const existingGlCodes: Set<string> = new Set(
  generalLedgerAccounts.data.map((gl) => gl.data.code)
)
const missingGlCodeLines: Map<string, number[]> = new Map()

for (let i = 0; i < allRows.length; i++) {
  const glCode = cleanValue(allRows[i]['GL Code'])
  if (!existingGlCodes.has(glCode)) {
    const lineNumber = i + 2  // +1 for 1-indexed, +1 for header row
    if (!missingGlCodeLines.has(glCode)) {
      missingGlCodeLines.set(glCode, [])
    }
    missingGlCodeLines.get(glCode).push(lineNumber)
  }
}

If any GL codes are missing, the entire import aborts — with a detailed error listing every invalid code and the exact line numbers where they appear:

if (missingGlCodeLines.size > 0) {
  for (const [code, lines] of missingGlCodeLines.entries()) {
    naoLogger.error(
      `Missing GL code: "${code}" — found on ${lines.length} rows: lines ${lines.join(', ')}`
    )
  }
  throw naoFormatErrorById('bad_request', {
    reason: `Import aborted — ${missingGlCodeLines.size} GL codes not found: ${[...missingGlCodeLines.keys()].join(', ')}`
  })
}

This is an all-or-nothing gate. The pipeline doesn't import 9,950 valid rows and skip 50 bad ones — that would leave your ledger in a partial state. It validates everything first, then processes everything. Fail before you begin, not while you're halfway through.

Step 7: Group and Transform​

CSV rows are flat. Journal entries are compound documents with a header and multiple line items. The grouping step bridges that gap:

const groupedTransactions = naoUtils._.groupBy(
  allRows, (row: any) => cleanValue(row['Trans #'])
)

Each group becomes one journal entry. Within the group, every row is transformed into a typed journal item line. This is where raw strings become domain objects:

// -->Parse: amounts — strip US thousands commas then convert to number
const debitAmount = naoUtils._.toNumber(
  (debitRaw || '0').replace(/,/g, '')
) || 0
const creditAmount = naoUtils._.toNumber(
  (creditRaw || '0').replace(/,/g, '')
) || 0

Dates get the same treatment — string to DateTime object via naoDateTime(), never new Date():

journalEntryDate: naoDateTime(date).toJSDate(),
accountingDate: naoDateTime(date).toJSDate(),

Every field is explicitly converted to its target type. Strings become numbers. Date strings become Date objects. GL codes become document references via lookup. The CSV column Class becomes a warehouseId through a resolution function. Nothing arrives at the database in its original CSV form.

Step 8: Write — The Easy Part​

After all that preparation, the actual write is anticlimactic:

await bob.executeRequestOrThrow(
  eventOptions.createJournalEntryWithItemsActionCfpQuery,
  payload,
  { skipSession: true, skipEvent: true }
)
countCreated++

One call per journal entry. The skipSession: true flag is deliberate — each entry auto-commits independently, avoiding MongoDB's 60-second transaction timeout on long-running imports. The skipEvent: true prevents downstream event cascades during bulk import.

Notice what's not here: no validation, no deduplication, no type conversion. All of that was handled in steps 1 through 7. The write trusts the pipeline above it.

The Preferred Alternative: flowBulk​

The executeRequestOrThrow approach works here because each journal entry is a compound document — a header with nested line items — and the downstream hook (create-data-entry-for-journal-entry) handles that compound creation logic. We're delegating to a hook that knows how to build both the parent and child documents atomically.

But for simpler, flat document imports — GL accounts, contacts, products, warehouses — flowBulk is the preferred write strategy. It batches all operations into a single database round-trip, which is dramatically faster than sequential per-document calls.

Here's what the write step looks like with flowBulk:

// -->Get: the flow collection
const flowCollection = bob.flowGlobal.getFlowCollection(eventOptions.glNaoQueryOptions)

// -->Init: bulk orders
const flowBulk = flowCollection.docs.flowQuery()
  .user(bob.flowUser)
  .flowOptions(eventOptions.glNaoQueryOptions)
  .bulkOrders()

// -->Build: bulk insert operations from prepared payloads
const flowQuery = flowCollection.docs.flowQuery()
  .user(bob.flowUser)
  .flowOptions(eventOptions.glNaoQueryOptions)

for (const account of preparedAccounts) {
  const flowData = flowQuery.flowData().setData(account)
  flowBulk.insertOne(flowData)
}

// -->Execute: all operations in a single batch
if (flowBulk.length > 0) {
  data = await flowCollection.docs.runBulk(
    bob.flowUser, { flowBulk }, bob.dbSession()
  )
}

The performance difference is significant. Importing 10,000 GL accounts with executeRequestOrThrow in a loop takes ~500 seconds (50ms × 10,000). The same import with flowBulk takes ~2 seconds — one database round-trip for the entire batch.

When to Use Which​

The journal entry import in this article uses executeRequestOrThrow because the create-data-entry-for-journal-entry hook does more than just insert — it creates the journal header, generates line items, calculates balances, and links to fiscal periods. That logic already exists and is tested. Duplicating it inside the import hook would be a mistake.

Use flowBulk when you can. Fall back to per-entry calls when the write involves business logic you shouldn't be reimplementing.

Step 9: Monitor and Report​

The hook tracks every outcome:

let countCreated = 0
let countSkipped = 0
let countDuplicate = 0
const skippedRows: { transNum: string; reason: string }[] = []

Memory usage is monitored every 30 seconds during long imports:

const memoryInterval = setInterval(() => {
  const mem = process.memoryUsage()
  naoLogger.log(
    `Memory — heap: ${(mem.heapUsed / 1024 / 1024).toFixed(1)}MB`
  )
}, 30_000)

Progress is logged every 500 transactions. And the final result gives you the full picture:

data = { countCreated, countSkipped, countDuplicate, skippedRows }

No silent failures. No "import complete" with no details. Every row is accounted for.

The Pipeline at a Glance​

Eight steps of preparation. One step of writing. That ratio is not an accident — it's the architecture.

Hard-Won Lessons From Production Imports​

The pipeline above works. But getting it to that state involved months of production imports, failed runs, angry rollbacks, and late-night debugging sessions. These are the lessons that don't show up in the code — they show up in the decisions behind it.

1. AI Doesn't Know Your Domain — And It Will Guess Confidently​

Data imports are soaked in domain knowledge. What does "Class" mean in this CSV? It's a warehouse. What's a Trans #? It's a grouping key for compound journal entries. What does a zero-amount line mean — is it an error, or is it a tax-exempt placeholder?

AI will generate a perfectly structured import hook. It will parse, loop, and write. But it will consistently get the meaning wrong because it doesn't know your chart of accounts, your fiscal calendar, your warehouse naming conventions, or what "posted" means in your system versus the source system.

Every column in the CSV carries implicit business context that the AI has never seen. The GL Code column isn't just a string — it's a foreign key into your chart of accounts, and the mapping between the source system's codes and yours might not be 1:1. The Date column isn't just a date — it determines which fiscal period the entry lands in, which affects financial reporting.

Expect to rewrite 50-70% of the AI's field mapping and business logic. Use the AI for the pipeline structure — the download, parse, loop, write skeleton. Do the domain mapping yourself.

2. Data Preparation Should Be 70% of the Work​

If you're spending most of your time on the database write, you're building the pipeline backward. The write is trivial once the data is clean. The hard part — the part that takes 70% of your development time — is everything before it:

• Understanding the source file format (delimiters, encoding, quoting)
• Identifying and filtering junk rows (subtotals, headers, blank lines)
• Mapping source columns to target fields
• Converting types (string amounts with commas → numbers, date strings → Date objects)
• Resolving foreign keys (GL code → generalLedgerAccountId, Class → warehouseId)
• Grouping flat rows into compound documents

If your commit history shows three days on data prep and an afternoon on the write, you're doing it right. If it's the inverse, your pipeline will break on the first real file.

3. The Error Rate Must Be Zero​

Not "low." Not "acceptable." Zero.

Financial data imports don't have a margin for error. If you import 10,000 journal entries and 3 are wrong, you don't have 99.97% accuracy — you have a corrupted ledger. The accountant won't find those 3 bad entries. They'll find them in six months during an audit, and by then the cascade of dependent transactions makes the fix exponentially harder.

This is why the pipeline uses an all-or-nothing preflight gate. If a single GL code is invalid, the entire import aborts. It's aggressive, and users sometimes push back — "just skip the bad ones" — but partial imports create partial ledgers, and partial ledgers create real financial risk.

Design your pipeline so it either completes perfectly or doesn't start at all.

4. Prepare the Workspace for Fast Import-Wipe Cycles​

You will run the import multiple times before it's right. The first run will reveal data issues you didn't anticipate. The second run will expose edge cases in your transformation logic. The third run might actually work.

This means you need the ability to wipe and reimport quickly:

• Tag all imported documents with a recognizable prefix (PIPE- in this hook) so you can query and delete them in bulk
• Use skipSession: true so each document auto-commits — if you need to kill a run midway, you know exactly what was written
• Keep the source file immutable — download it once, process it from local storage, never modify the original
• Have a cleanup script ready that deletes all PIPE-* journal entries so you can re-run from scratch

The import-wipe-reimport cycle is your development loop. Make it fast. If a wipe takes 20 minutes, you'll only test three iterations in an hour. If it takes 30 seconds, you'll test twenty.

5. The Audit Trail Checklist​

Before writing the first line of code, answer these questions:

• Naming convention: How will imported documents be identified? (PIPE-{transNum}, IMPORT-{batchId}, etc.)
• Deduplication strategy: What makes two entries "the same"? Name? Source ID? Content hash?
• Audit fields: Does each imported document carry its source file reference, import timestamp, and batch ID?
• Transaction grouping: Are compound documents (header + lines) created atomically or individually?
• Batching strategy: One DB call per document, or batched inserts? Does the downstream hook support batch payloads?
• Session management: Shared transaction across all writes, or independent auto-commits? (For long imports, independent commits avoid MongoDB's 60-second transaction timeout)
• Event suppression: Should downstream events (notifications, recalculations) fire during import, or be suppressed with skipEvent: true?
• Rollback plan: If the import is wrong, how do you delete everything it created?
• Completion guarantee: If the process crashes at row 5,000 of 10,000, can you resume or must you wipe and restart?

Answer every one of these before you start coding. They shape the pipeline's architecture in ways that are expensive to change later.

6. Measure Every Prep Query — You'll Be Surprised​

Every reference data query in the prep stage loads data into memory. You need to know how much. The instinct is to worry about "200,000 documents" — that sounds like a lot. But if you're only projecting the name field, 200,000 short strings might be 2MB. That's nothing.

On the other hand, fetching 10,000 full documents with nested arrays of line items could be 500MB.

Measure, don't guess:

const result = await fc.docs.flowQuery()
  .projection({ 'data.name': 1 })
  .limit(2_000_000)
  .getMany()

// Log the actual size
const sizeInBytes = Buffer.byteLength(JSON.stringify(result.data))
naoLogger.log(`Dedup index: ${result.data.length} docs, ${(sizeInBytes / 1024 / 1024).toFixed(1)}MB`)

This changes your architecture decisions. If the dedup index is 2MB, load it all into a Set — fast and safe. If it's 200MB, you need a different strategy: streaming comparison, database-side dedup, or chunked lookups.

The projection operator is your best friend here. The difference between { 'data.name': 1 } and fetching full documents can be 100x in memory footprint. Always project to the minimum fields you need, and always log the actual size.

7. There Is No Room for Error — The Import Must Complete​

A 50% imported ledger is worse than a 0% imported ledger. With zero imports, the customer knows nothing happened. With half the data imported, they don't know which half — and they can't trust any of it.

This has design implications:

• Preflight everything: Validate the entire file before writing the first document. If row 9,999 has an invalid GL code, you want to know before row 1 is committed.
• Idempotent design: If the process crashes and restarts, it should skip already-imported entries (the dedup Set) and continue — not create duplicates.
• Progress logging: Log every 500 transactions so you can see exactly where a failure occurred. The memory monitor isn't vanity — it's your early warning system for OOM kills.
• Independent commits: skipSession: true means each journal entry is its own commit. If the process dies at entry 5,001, entries 1–5,000 are safe and the restart picks up at 5,001.

Design for the crash. It will happen. The question is whether you recover in minutes or spend a weekend rebuilding the ledger.

8. The Validation Report Is Your Customer Deliverable​

Here's a pattern that saves weeks of back-and-forth: run the pipeline in validation-only mode first.

The preflight validation step — the one that checks GL codes, resolves warehouses, and verifies fiscal periods — produces a report that is exactly what the customer needs to clean their data. Don't just throw an error. Format the validation results as a deliverable:

Import Validation Report
========================
File: general-ledger-export-2024.csv
Total rows: 12,847
Valid rows: 12,691
Skipped (no Trans #): 142
Invalid GL codes: 3
  - "4510" → found on 8 rows (lines 234, 567, 891, ...)
  - "6200-A" → found on 2 rows (lines 1002, 1003)
  - "MISC" → found on 4 rows (lines 5501, 5502, 5503, 5504)

Action required: Add missing GL codes to chart of accounts, or update the CSV to use valid codes.

Send this to the customer before running the actual import. Nine times out of ten, this report is all they need. They fix the three GL codes in their spreadsheet, re-export, and the next run succeeds cleanly.

The validation stage isn't just a safety gate — it's a communication tool. It speaks the customer's language (GL codes, line numbers) instead of yours (stack traces, error IDs).

9. Always Benchmark at Scale — What If This File Has 10 Million Entries?​

Your test file has 500 rows and runs in 3 seconds. Your production file has 200,000 rows. Your largest customer's file has 10 million. Does your pipeline survive?

Things that break at scale:

• Memory: Loading 10 million rows into a JavaScript array. At ~1KB per parsed row object, that's 10GB. Your 8GB server is dead.
• Dedup index: A Set of 2 million strings is fine (~150MB). A Set of 50 million strings is not.
• Sequential writes: 10 million executeRequestOrThrow calls at 50ms each = 139 hours. You need batching or parallelism.
• Logging: Logging every skipped row with its reason — if 2 million rows are skipped, your skippedRows array consumes more memory than the actual data.
• Progress intervals: Logging every 500 transactions makes sense at 10,000. At 10 million, that's 20,000 log lines — still manageable, but worth verifying.

Run the numbers before you run the import:

If the 10M column has red marks, your architecture needs to change — streaming parsers, chunked processing, parallel writes, or database-side dedup. Don't discover this in production.

The file you're testing with is never the file you'll receive.

The Takeaway​

The instinct is to start with the database write and work backward. Production import pipelines work the other way: start with the source data and build forward through layers of validation, cleaning, and transformation until the write becomes a formality.

Data preparation is 70% of the work. The error rate must be zero. The AI will get the structure right and the domain wrong. Your validation report is your customer's favorite deliverable. And the file you tested with is never the file you'll receive in production.

Your CSV is not your data. It's raw material. The pipeline is the refinery.

Trust the data only after you've earned that trust — one step at a time.

AI coding assistants are remarkably good at following patterns. They're remarkably bad at knowing when a pattern doesn't apply. Here are three categories where you should always take the wheel.

The Premise​

AI generates code by pattern matching against its training data and your project's skill files. For 80% of Logic Bee hooks, this produces correct, convention-following code. But certain categories of logic require reasoning that current AI models consistently get wrong — not because they're broken, but because the problems require domain knowledge that never appears in code patterns alone.

The Story​

A senior developer tracked every AI-generated hook over three months and categorized the corrections. Three patterns emerged: financial calculations, multi-tenant edge cases, and transaction session management. Each required the same type of intervention — the developer understanding why, not just how.

Pattern 1: Financial Calculations​

What the AI Does​

AI uses JavaScript arithmetic because it looks correct:

// ❌ AI-generated: floating point disaster
const taxAmount = subtotal * taxRate
const total = subtotal + taxAmount + shippingCost

The problem isn't obvious until you see: 199.99 * 0.08 = 15.999200000000001.

What the Developer Does​

// ✅ mathChain for precision
const taxAmount = naoUtils.mathChain(subtotal).multiply(taxRate).done()
const total = naoUtils.mathChain(subtotal)
  .add(taxAmount)
  .add(shippingCost)
  .done()

Why AI Gets This Wrong​

AI sees * and + operators in countless training examples. mathChain() is Logic Bee-specific. Even with skill files telling the AI to use mathChain, it sometimes falls back to raw operators for "simple" calculations — but there's no such thing as a simple financial calculation.

Rule: Any arithmetic involving currency → naoUtils.mathChain(). No exceptions.

Pattern 2: Multi-Tenant Edge Cases​

What the AI Does​

AI handles the happy path correctly — querying documents with .user(bob.flowUser). But it fails on edge cases where tenant boundaries interact in unexpected ways:

// ❌ AI-generated: cross-tenant data leak in batch processing
const allVendors = await vendorCollection.docs.flowQuery()
  .user(bob.flowUser)
  .flowOptions(eventOptions.vendorNaoQueryOptions)
  .query({ 'data.type': 'preferred' })
  .getMany(undefined, 'PurchasingInterface.Vendor')

// AI then uses vendor IDs to query another collection
// But what if a vendor was shared across tenants?
const vendorBills = await billCollection.docs.flowQuery()
  .user(bob.flowUser)
  .flowOptions(eventOptions.billNaoQueryOptions)
  .query({ 'data.vendorId': { $in: allVendors.map(v => v.docId) } })
  .getMany(undefined, 'FinanceInterface.Bill')

What the Developer Does​

The developer adds business-unit scoping:

// ✅ Add explicit business unit filtering
const vendorBills = await billCollection.docs.flowQuery()
  .user(bob.flowUser)
  .flowOptions(eventOptions.billNaoQueryOptions)
  .query({
    'data.vendorId': { $in: allVendors.map(v => v.docId) },
    'data.businessUnitId': bob.flowUser.getBusinessUnitId()  // explicit scope
  })
  .getMany(undefined, 'FinanceInterface.Bill')

Why AI Gets This Wrong​

Multi-tenancy is an architectural concern, not a code pattern. The AI knows to add .user(bob.flowUser) because the skill file says so. But it doesn't understand why — that in a shared-vendor scenario, FlowQuery's default scoping may not be sufficient and additional business-unit filtering is needed.

Rule: Any cross-collection join → verify tenant and business-unit boundaries manually.

Pattern 3: Complex Transaction Sessions​

What the AI Does​

AI includes bob.dbSession() on write operations because the skill file says so. But it doesn't understand transaction design:

// ❌ AI-generated: partial transaction safety
try {
  // Operation 1: create credit memo
  await creditMemoCollection.docs.flowQuery()
    .user(bob.flowUser)
    .flowOptions(eventOptions.creditMemoNaoQueryOptions)
    .insertOne(creditMemoDoc, bob.dbSession())

  // Operation 2: update invoice status
  await invoiceCollection.docs.flowQuery()
    .user(bob.flowUser)
    .flowOptions(eventOptions.invoiceNaoQueryOptions)
    .docId(invoiceId)
    .updateOne({ 'data.status': 'credited' }, bob.dbSession())

  // Operation 3: adjust vendor balance (NO SESSION!)
  await vendorCollection.docs.flowQuery()
    .user(bob.flowUser)
    .flowOptions(eventOptions.vendorNaoQueryOptions)
    .docId(vendorId)
    .updateOne({ 'data.balance': newBalance })  // ← missing bob.dbSession()
}

What the Developer Does​

The developer ensures all writes within a logical transaction share the session — and adds rollback awareness:

// ✅ All writes in the same session, with failure awareness
const session = bob.dbSession()

await creditMemoCollection.docs.flowQuery()...insertOne(creditMemoDoc, session)
await invoiceCollection.docs.flowQuery()...updateOne({ 'data.status': 'credited' }, session)
await vendorCollection.docs.flowQuery()...updateOne({ 'data.balance': newBalance }, session)

Why AI Gets This Wrong​

The AI treats bob.dbSession() as a "should include" parameter, not a "must include for atomicity" requirement. It doesn't reason about what happens when operation 3 fails — operation 1 and 2 commit, but the vendor balance is stale.

Rule: If you're touching more than one collection → every write operation must use bob.dbSession(). Audit this manually every time.

The Decision Framework​

The Takeaway​

The AI is your fastest pair programmer — for patterns it has seen before. For patterns that require understanding, not just matching, the developer must intervene.

Three questions to ask about every AI-generated hook:

1. Is there money involved? → Verify all math uses mathChain()
2. Are there cross-collection queries? → Verify tenant and business-unit scoping
3. Are there multiple writes? → Verify every one uses bob.dbSession()

Know when to let the AI drive. Know when to take the wheel.

Importing 10,000 general ledger accounts shouldn't require 10,000 lines of code. Here's how we built a complete data import pipeline with AI — and the six places where the developer had to step in.

The Premise​

Data import is one of ERP's most common, tedious, and error-prone workflows. Parse a file, validate rows, map fields, handle duplicates, write to the database, and report errors — all with transaction safety and tenant isolation. It's the perfect candidate for AI generation because the structure is repetitive, but the business logic is unique per import type.

The Story​

The team needed to import general ledger (GL) accounts from a client's CSV export. The import pipeline needed to:

1. Parse a CSV file with 10,000+ rows
2. Validate each row against a GL account schema
3. Skip duplicate account numbers
4. Create new accounts in the finance/general-ledger collection
5. Report successes, skips, and failures in a structured response

A developer prompted the AI to build the entire pipeline. The AI generated 90% of the working code. But six critical interventions turned it from "mostly working" into "production-ready."

The Pipeline Architecture​

CSV Upload → Parse → Validate → Deduplicate → Insert → Report

Each step maps to a section of the hook:

@LogicHook({
  name: 'ImportPipeGeneralLedgerAccounts',
  path: 'import-pipe/general-ledger-accounts',
  library: 'import-pipe',
  method: 'general-ledger-accounts',
  legacy: 'app/app.importPipeHooks.generalLedgerAccounts'
})

Where the Developer Stepped In​

Intervention 1: CSV Parsing Edge Cases​

The AI used a basic split-by-comma approach. Real CSV files have quoted fields, embedded commas, and encoding issues.

// ❌ AI-generated: naive parsing
const rows = csvContent.split('\n').map(line => line.split(','))

// ✅ Developer fix: use the project's CSV utility
const rows = naoUtils.csv.parse(csvContent, {
  header: true,
  skipEmptyLines: true,
  dynamicTyping: false  // keep everything as strings for validation
})

Intervention 2: Batch Size for Database Operations​

The AI generated a loop that inserted one document at a time — 10,000 individual insert operations.

// ❌ AI approach: one-by-one inserts
for (const account of validAccounts) {
  await fc.docs.flowQuery()
    .user(bob.flowUser)
    .flowOptions(eventOptions.glNaoQueryOptions)
    .insertOne(account, bob.dbSession())
}

// ✅ Developer fix: batch inserts
const BATCH_SIZE = 500
for (let i = 0; i < validAccounts.length; i += BATCH_SIZE) {
  const batch = validAccounts.slice(i, i + BATCH_SIZE)
  await fc.docs.flowQuery()
    .user(bob.flowUser)
    .flowOptions(eventOptions.glNaoQueryOptions)
    .insertMany(batch, bob.dbSession())
}

Intervention 3: Duplicate Detection Strategy​

The AI checked for duplicates by querying each account number individually. The developer replaced it with a single bulk query:

// ✅ Fetch all existing account numbers in one query
const existing = await fc.docs.flowQuery()
  .user(bob.flowUser)
  .flowOptions(eventOptions.glNaoQueryOptions)
  .query({ 'data.accountNumber': { $in: incomingAccountNumbers } })
  .project({ 'data.accountNumber': 1 })
  .getMany(undefined, 'FinanceInterface.GeneralLedgerAccount')

const existingSet = new Set(existing.map(e => e.data.accountNumber))
const newAccounts = validAccounts.filter(a => !existingSet.has(a.data.accountNumber))

Intervention 4: Validation Schema​

The AI generated a validation schema that was too permissive — accepting any string for account type. The developer added enum constraints:

// ✅ Strict validation
const accountSchema = SuperJoi.object({
  accountNumber: SuperJoi.string().pattern(/^\d{4,6}$/).required(),
  accountName: SuperJoi.string().max(200).required(),
  accountType: SuperJoi.string().valid(
    'asset', 'liability', 'equity', 'revenue', 'expense'
  ).required(),
  parentAccount: SuperJoi.string().allow(null, ''),
  isActive: SuperJoi.boolean().default(true)
})

Intervention 5: Error Reporting Granularity​

The AI returned a generic "X succeeded, Y failed" message. The developer added row-level error reporting:

// ✅ Row-level reporting
const report = {
  total: rows.length,
  created: newAccounts.length,
  skipped: existingSet.size,
  errors: validationErrors.map(e => ({
    row: e.rowNumber,
    accountNumber: e.data?.accountNumber || 'unknown',
    reason: e.error.message
  }))
}

Intervention 6: Memory Management​

With 10,000+ rows, the AI was holding all raw data, parsed data, and results in memory simultaneously. The developer added streaming-style processing:

// ✅ Process in chunks, release references
let processedCount = 0
for (let i = 0; i < rows.length; i += BATCH_SIZE) {
  const chunk = rows.slice(i, i + BATCH_SIZE)
  const { valid, errors } = validateChunk(chunk, i)
  validationErrors.push(...errors)
  const newInChunk = valid.filter(a => !existingSet.has(a.data.accountNumber))
  if (newInChunk.length > 0) {
    await insertBatch(fc, bob, newInChunk, eventOptions)
    processedCount += newInChunk.length
  }
  // chunk goes out of scope, eligible for GC
}

The Final Result​

The Takeaway​

AI generates excellent pipeline structure. The developer adds:

1. Domain-specific parsing — real-world files aren't clean
2. Performance optimization — batch operations, bulk queries
3. Strict validation — business rules the AI doesn't know
4. Detailed reporting — users need row-level feedback
5. Resource management — memory and connection awareness

The AI builds the highway. You paint the lane markings.

The hook runs. It returns ok: true. The data looks right. But three hooks later in the pipeline, something breaks — and the stack trace points nowhere useful. Welcome to debugging in an event-driven system.

The Premise​

Debugging AI-generated hooks is different from debugging hand-written code. The code looks correct — it follows every convention, passes validation, and compiles cleanly. But AI optimizes for pattern matching, not for understanding the execution context. When a bug hides in the space between hooks, traditional debugging falls apart.

The Story​

A developer asks the AI to generate a hook for processing refunds. The hook queries the original invoice, calculates the refund amount, creates a credit memo, and updates the invoice status. The AI generates clean code. Tests pass. But in production, customers report that their invoice status shows "refunded" even when the credit memo creation fails.

The root cause? The AI wrote the status update before the credit memo creation — and without a shared database session, the two operations weren't atomic.

The Debugging Playbook​

Step 1: Trace the Event Route​

Every request in Logic Bee flows through: HTTP → Auth → FlowUser → Event Engine → Hook(s) → Response. When something goes wrong, first verify that the right hook is even being called:

// Add temporary logging at the top of the try block
console.log(`[DEBUG] ${new Date().toISOString()} - Hook entered`, {
  hookPath: 'finance-bills/process-refund',
  docId: bob.dataPayload?.docId,
  userId: bob.flowUser?.getUserInfo()?.id
})

If you don't see this log, the event routing is wrong — check the @LogicHook decorator's path property.

Step 2: Verify the BobRequest Lifecycle​

The bob object carries context through the entire execution. AI-generated code sometimes breaks the lifecycle by:

• Modifying bob.dataPayload directly instead of using bob.replaceDataPayload()
• Not passing bob.dbSession() to write operations, breaking transaction boundaries
• Calling bob.executeRequestOrThrow() without error checking

Check the bob state at each step:

// -->Debug: log bob state before critical operations
console.log('[DEBUG] dataPayload:', JSON.stringify(bob.dataPayload, null, 2))
console.log('[DEBUG] dbSession exists:', !!bob.dbSession())

Step 3: Check Operation Order​

AI generates operations in the order they appear in the prompt, which may not be the correct execution order. In the refund story, the fix was:

// ❌ AI-generated order (status before credit memo)
await updateInvoiceStatus(bob, 'refunded')
await createCreditMemo(bob, refundData)

// ✅ Correct order (credit memo first, then status)
const creditMemo = await createCreditMemo(bob, refundData)
if (creditMemo) {
  await updateInvoiceStatus(bob, 'refunded')
}

Step 4: Verify Transaction Boundaries​

When a hook performs multiple writes, all should share the same database session:

// ❌ AI sometimes creates independent write operations
await collection1.docs.flowQuery()...updateOne({ status: 'refunded' })
await collection2.docs.flowQuery()...insertOne(creditMemoDoc)

// ✅ Both must use bob.dbSession()
await collection1.docs.flowQuery()...updateOne({ status: 'refunded' }, bob.dbSession())
await collection2.docs.flowQuery()...insertOne(creditMemoDoc, bob.dbSession())

Step 5: Check Sub-Request Error Propagation​

When hooks call other hooks via bob.executeRequestOrThrow() or bob.runAction(), AI sometimes ignores the returned error:

// ❌ AI ignores sub-request failure
const result = await bob.runAction('inventory.adjust-stock', { data: adjustmentData })
// continues execution even if result.ok === false

// ✅ Check the result
const result = await bob.runAction('inventory.adjust-stock', { data: adjustmentData })
if (!result.ok) {
  throw result.error || naoFormatErrorById('bad_request', {
    reason: 'Stock adjustment failed during refund processing'
  })
}

Common AI-Generated Bug Patterns​

The Takeaway​

When debugging AI-generated hooks in a pipeline:

1. Start at the route — is the right hook being called?
2. Check the bob lifecycle — is the request context intact?
3. Verify operation order — does the sequence match the business requirement?
4. Audit every write — does each one use bob.dbSession()?
5. Test sub-requests — are errors from child hooks handled?

The AI writes code that looks right. Your job is to verify it runs right — in context, in sequence, and under failure.

AI follows the rules it knows. Your job is knowing the rules it missed. Here's the definitive review checklist for AI-generated Logic Bee hooks — 10 hard rules and 12 soft conventions, explained with real examples.

The Premise​

Logic Bee's AI validation system checks generated hooks against two categories of rules: hard rules that must never be violated, and soft conventions that should be followed unless there's a documented reason not to. The AI self-validates, but no validation is perfect — and understanding why each rule exists makes you a better reviewer.

The Story​

A junior developer merges an AI-generated hook without review. It passes all automated checks. Two weeks later, a production incident reveals that the hook was silently mutating bob.dataPayload directly — causing downstream hooks in the pipeline to receive corrupted data. The mutation wasn't caught because it's a soft convention violation, not a hard rule.

That incident inspired this checklist.

The 10 Hard Rules​

These are non-negotiable. If any of these fail, the hook must not be merged.

1. ok Must Be Assigned Before Return​

// ❌ Missing ok assignment in success path
try {
  data = await someOperation()
  // forgot: ok = true
} catch (err) { error = err; ok = false }

2. data Must Be Assigned Before Return​

Every successful execution must set data — even if it's just { success: true }.

3. bob.flowUser Must Be Passed to Every FlowQuery​

No exceptions. No helper functions that skip it. Every .flowQuery() chain needs .user(bob.flowUser).

4. returnEventResult Must Be the Last Statement​

// ❌ Code after returnEventResult
return returnEventResult(bob, { ok, error, data })
console.log('done')  // unreachable but indicates confusion

5. Error Variable Must Be Set in Catch Block​

// ❌ Missing error assignment
catch (err) {
  ok = false
  // forgot: error = err
}

6. try/catch Must Wrap All Logic​

No business logic outside the try block except variable initialization.

7. FlowQuery Must Include .flowOptions()​

Every query needs the collection metadata to resolve correctly.

8. Decorator Metadata Must Match File Path​

path: 'finance-bills/calculate-late-fees' must correspond to the actual file location.

9. Write Operations Must Pass bob.dbSession()​

For transactional integrity across multi-document updates.

10. Thrown Errors Must Use naoFormatErrorById​

Raw throw new Error() produces unstructured error responses.

The 12 Soft Conventions​

These improve code quality and consistency. Violations should be fixed unless there's a strong reason to deviate.

1. eventOptions First​

Declare the eventOptions object as the first statement inside the try block.

try {
  const eventOptions = {
    billNaoQueryOptions: { docName: 'bill', cfpPath: 'finance/bills' }
  }
  // ... rest of logic
}

2. Arrow Comments for Structure​

Use // -->Get:, // -->Set:, // -->Validate:, // -->Iterate:, // -->Update: prefix comments.

3. naoUtils.mathChain() for Math​

Never use raw arithmetic for financial calculations. Floating point errors are real.

// ❌ Raw arithmetic
const total = quantity * unitPrice

// ✅ Math chain
const total = naoUtils.mathChain(quantity).multiply(unitPrice).done()

4. naoDateTime() for Dates​

Never use new Date() or Date.now(). The Luxon-based wrapper handles timezones correctly.

5. Don't Mutate bob.dataPayload​

Read from it. Don't write to it. Use bob.replaceDataPayload() if you need to pass modified data downstream.

6. One Responsibility Per Hook​

If the AI generates a hook that both validates and processes, consider splitting it into two hooks.

7. Guard Clauses Before Logic​

Check for nulls, empty arrays, and invalid states at the top of the try block.

8. Descriptive Error Reasons​

// ❌ Vague
throw naoFormatErrorById('bad_request', { reason: 'Invalid' })

// ✅ Descriptive
throw naoFormatErrorById('bad_request', {
  reason: `Bill ${bill.docId} has zero balance and cannot accrue late fees`
})

9. TypeScript Interface References​

Pass the interface name to .getMany() and .getOne() for type safety.

10. Consistent Variable Naming​

Use fc for flow collections, doc for single documents, docs for arrays.

11. No console.log in Production Code​

Use the structured logging utilities from @logic-bee/utils.

12. End With ok = true; data = result​

The last two lines of the try block should always be the success assignment.

The Review Shortcut​

When reviewing AI-generated hooks, scan in this order:

1. Scroll to the bottom — is returnEventResult the last statement?
2. Search for .flowQuery() — does every chain have .user(bob.flowUser)?
3. Search for .updateOne() / .updateMany() — does every write pass bob.dbSession()?
4. Read the catch block — are both error = err and ok = false present?
5. Read the first line of the try block — is eventOptions declared first?

If all five pass, you're likely looking at a clean hook.

The Takeaway​

Print this. Pin it next to your monitor. Use it every time you review AI output:

Hard rules = merge blockers. Soft conventions = code quality. Both = production readiness.

Every Stripe integration starts the same way — auth headers, request builders, error mappers, retry logic. Let the AI handle the boilerplate so you can focus on the business logic that makes your integration unique.

The Premise​

Logic Bee's connector architecture separates external API communication into reusable connector classes in libs/connectors/. When a hook needs to talk to Stripe, QuickBooks, or UPS, it calls a connector — never the raw HTTP client. This pattern is perfect for AI generation because connectors follow a rigid contract.

The developer's role shifts from writing HTTP calls to validating auth flows, error mapping, and retry behavior.

The Story​

The team needs a new ShipStation integration for the shipping-management library. A developer prompts:

"Add a ShipStation connector that creates shipment labels and tracks packages. Wire it into a new hook called create-shipment-label in shipping-management. Use the connectors, creating-hooks, and bob-request-api skills."

The AI generates two things: the connector class and the hook that uses it.

What the AI Generates​

The Connector​

export class ShipStationConnector {
  private baseUrl = 'https://ssapi.shipstation.com'
  
  constructor(private apiKey: string, private apiSecret: string) {}

  async createLabel(shipmentData: CreateLabelPayload): Promise<ShipStationLabel> {
    const response = await fetch(`${this.baseUrl}/shipments/createlabel`, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Authorization': `Basic ${Buffer.from(`${this.apiKey}:${this.apiSecret}`).toString('base64')}`
      },
      body: JSON.stringify(shipmentData)
    })
    
    if (!response.ok) {
      throw naoFormatErrorById('bad_request', {
        reason: `ShipStation API error: ${response.status}`
      })
    }
    
    return response.json()
  }
}

The Hook​

@LogicHook({
  name: 'ShippingManagementCreateShipmentLabel',
  path: 'shipping-management/create-shipment-label',
  library: 'shipping-management',
  method: 'create-shipment-label',
  legacy: 'app/app.shippingManagementHooks.createShipmentLabel'
})
export class ShippingManagementCreateShipmentLabel {
  public static execute() {
    return async (bob: BobRequest<{ data: { data: any; naoQueryOptions: NaoQueryOptions } }>) => {
      let ok = true, error: any = null, data
      try {
        const eventOptions = {
          shipmentQuery: { docName: 'shipment', cfpPath: 'shipping/shipments' }
        }

        // -->Get: shipment document
        const shipment = await bob.flowGlobal
          .getFlowCollection(eventOptions.shipmentQuery)
          .docs.flowQuery()
          .user(bob.flowUser)
          .flowOptions(eventOptions.shipmentQuery)
          .docId(bob.dataPayload.docId)
          .getOne()

        if (!shipment) {
          throw naoFormatErrorById('not_found', { reason: 'Shipment not found' })
        }

        // -->Create: shipping label via connector
        const connector = new ShipStationConnector(
          process.env.SHIPSTATION_API_KEY,
          process.env.SHIPSTATION_API_SECRET
        )
        const label = await connector.createLabel({
          orderId: shipment.data.orderId,
          weight: shipment.data.totalWeight,
          dimensions: shipment.data.dimensions,
          shipTo: shipment.data.shippingAddress
        })

        // -->Update: shipment with tracking info
        await bob.flowGlobal
          .getFlowCollection(eventOptions.shipmentQuery)
          .docs.flowQuery()
          .user(bob.flowUser)
          .flowOptions(eventOptions.shipmentQuery)
          .docId(shipment.docId)
          .updateOne({
            'data.trackingNumber': label.trackingNumber,
            'data.labelUrl': label.labelUrl,
            'data.status': 'label-created'
          }, bob.dbSession())

        ok = true
        data = { docIds: [shipment.docId], trackingNumber: label.trackingNumber }
      } catch (err) { error = err; ok = false }
      return returnEventResult(bob, { ok, error, data })
    }
  }
}

What You Need to Verify​

1. Credential Management​

AI often hardcodes process.env.VARIABLE_NAME directly. In Logic Bee, credentials should flow through the configuration layer:

// ❌ AI default
const connector = new ShipStationConnector(
  process.env.SHIPSTATION_API_KEY,
  process.env.SHIPSTATION_API_SECRET
)

// ✅ Better: use workspace-scoped config
const config = bob.flowUser.getWorkspaceConfig('shipstation')
const connector = new ShipStationConnector(config.apiKey, config.apiSecret)

2. Error Response Mapping​

The AI generates a generic error handler. Real APIs return structured errors — map them:

// ✅ Map API-specific errors to Logic Bee error format
if (!response.ok) {
  const errorBody = await response.json()
  throw naoFormatErrorById('bad_request', {
    reason: `ShipStation: ${errorBody.ExceptionMessage || response.statusText}`,
    details: errorBody
  })
}

3. Rate Limiting and Retries​

External APIs have rate limits. The AI rarely adds retry logic unless prompted.

4. Payload Validation​

Verify the connector validates outbound payloads before sending — missing required fields should fail fast, not at the API level.

The Takeaway​

The connector integration checklist:

• Credentials loaded from workspace config, not process.env
• API errors mapped to naoFormatErrorById with descriptive reasons
• Rate limiting and retry logic included
• Outbound payload validated before API call
• Response types match TypeScript interfaces
• API-specific edge cases documented in connector class

Let AI wire the plumbing. You verify the valves.

What if creating a new business logic hook took 5 minutes instead of 45? With AI-assisted scaffolding in Logic Bee, it can — but only if you know what to verify before hitting merge.

The Premise​

Every Logic Bee hook follows the same anatomy: a @LogicHook decorator, a static execute() method, the Bob Wrapper pattern, and a returnEventResult call. That's a lot of boilerplate that's identical across hundreds of hooks — and boilerplate is exactly what AI excels at.

But scaffolding is only half the story. The other half is knowing what the AI cannot know: your business rules, your tenant boundaries, and the edge cases that live in your head.

The Story​

A developer needs a new hook in the finance-bills library to calculate late fees on overdue invoices. Instead of copy-pasting from an existing hook, they open their AI agent and type:

"Create a new hook in finance-bills that calculates late fees on overdue invoices. Skip bills with zero balance. Use the creating-hooks and flow-query skills."

In under a minute, the AI:

1. Runs the CLI — npx tsx scripts/logic-bee-create-new-hook.ts scaffolds the folder, .hook.ts, hook.yml, and context.yaml
2. Searches for patterns — finds calculate-bill as a reference implementation
3. Writes the logic — generates a complete hook with FlowQuery chains, naoDateTime() comparisons, and naoUtils.mathChain() for fee calculations
4. Self-validates — checks all 10 hard rules and 12 soft conventions

The developer has working code. But is it production-ready?

What the AI Gets Right​

The generated hook follows every structural convention perfectly:

• File path: hooks/finance-bills/calculate-late-fees/calculate-late-fees.finance-bills.hook.ts
• Decorator metadata: name, path, library, method — all correctly derived
• Bob Wrapper: let ok = true, error: any = null, data → try/catch → returnEventResult
• Arrow comments: // -->Get:, // -->Set:, // -->Iterate:
• Error handling: naoFormatErrorById('bad_request', { reason: '...' })

This is the 80% that would have taken a human 30 minutes to type.

What You Need to Verify​

1. Tenant Scoping​

Every FlowQuery chain must pass bob.flowUser. The AI almost always includes it, but a missing .user(bob.flowUser) means cross-tenant data leakage.

// ✅ AI usually generates this correctly
const bills = await fc.docs.flowQuery()
  .user(bob.flowUser)  // ← verify this exists on EVERY query
  .flowOptions(eventOptions.billNaoQueryOptions)
  .query({ 'data.status': 'posted' })
  .getMany(undefined, 'FinanceInterface.Bill')

2. Business Logic Accuracy​

The AI calculated late fees as a flat percentage. Your business rule says it should be tiered: 1.5% for 1–30 days, 3% for 31–60 days, 5% for 60+. The AI doesn't know this unless you told it.

3. Transaction Session Usage​

For hooks that update multiple documents, verify bob.dbSession() is passed to all write operations. The AI may omit it on some update calls, breaking atomicity.

4. Edge Cases​

The prompt said "skip bills with zero balance," and the AI added a guard clause. But what about negative balances (credits)? What about bills in draft status?

The Takeaway​

AI scaffolding eliminates the mechanical overhead of creating hooks. Here's your pre-merge checklist:

• .user(bob.flowUser) on every FlowQuery chain
• bob.dbSession() on every write operation
• Business logic matches actual requirements (not AI assumptions)
• Edge cases covered: nulls, zero values, unexpected statuses
• eventOptions declared at the top of the try block

The rule of thumb: let the AI write the structure, but own the logic.