Skip to main content

Data Transformation with Functions

Functions allow you to transform, enrich, filter and validate your $event data in real-time before it reaches your data warehouse. Write custom JavaScript logic to modify $events, add derived fields, integrate external data, or filter out unwanted traffic.

What are Functions?​

Functions are JavaScript code snippets that execute on every $event flowing through your pipeline. They run in a secure, isolated sandbox between $event validation and warehouse delivery.

Data flow with Functions:

Source → Schema Validation → [Functions] → Destination

What You Can Do with Functions​

Transform

  • Modify field values
  • Restructure $event data
  • Normalize formats
  • Convert data types

Enrich

  • Add derived fields (calculations)
  • Add geo-location data from IP
  • Parse user agents
  • Lookup external data (APIs, databases)

Filter

  • Remove bot traffic
  • Filter test $events
  • Sample high-volume $events
  • Route $events conditionally

Validate

  • Apply custom business logic
  • Cross-field validation
  • Check data consistency

Function Basics​

Function Structure​

Every function receives an $event object and must return:

  • The modified $event to continue processing
  • null to drop the $event (won't be delivered)
function transform($event) {
  // Your transformation logic here

  return $event; // Continue processing
  // return null; // Drop $event
}

Available JavaScript Features​

Functions support ES6+ JavaScript with these additional support for network requests via fetch.

Not supported:

  • ❌ File system access
  • ❌ Global state/variables between executions
  • ❌ setTimeout/setInterval
  • ❌ require/import (use built-in utilities)

Common Transformation Patterns​

1. Add Derived Fields​

Calculate new fields from existing data:

function transform($event) {
  // Add tax calculation
  if ($event.amount) {
    $event.amount_with_tax = $event.amount * 1.08;
    $event.tax_amount = $event.amount * 0.08;
  }

  // Add is_high_value flag
  $event.is_high_value = $event.amount > 1000;

  // Add computed timestamp fields
  const eventDate = new Date($event.timestamp);
  $event.event_date = eventDate.toISOString().split('T')[0]; // "2024-01-15"
  $event.event_hour = eventDate.getUTCHours();
  $event.event_day_of_week = eventDate.getUTCDay(); // 0=Sunday

  return $event;
}

2. Data Normalization​

Standardize values and formats:

function transform($event) {
  // Normalize email to lowercase
  if ($event.email) {
    $event.email = $event.email.toLowerCase().trim();
  }

  // Normalize phone numbers (remove formatting)
  if ($event.phone) {
    $event.phone = $event.phone.replace(/[^0-9]/g, '');
  }

  // Normalize currency codes to uppercase
  if ($event.currency) {
    $event.currency = $event.currency.toUpperCase();
  }

  // Normalize country codes
  if ($event.country) {
    $event.country = $event.country.toUpperCase(); // US, GB, FR
  }

  // Round amounts to 2 decimals
  if ($event.amount) {
    $event.amount = Math.round($event.amount * 100) / 100;
  }

  return $event;
}

3. Data Enrichment​

Add context from existing fields:

function transform($event) {
  // Parse user agent for device info
  const ua = $event.context.user_agent || '';
  $event.device_type = /mobile/i.test(ua) ? 'mobile' :
                      /tablet/i.test(ua) ? 'tablet' : 'desktop';
  $event.is_mobile = /mobile/i.test(ua);

  // Extract browser
  if (/chrome/i.test(ua)) $event.browser = 'Chrome';
  else if (/safari/i.test(ua)) $event.browser = 'Safari';
  else if (/firefox/i.test(ua)) $event.browser = 'Firefox';
  else $event.browser = 'Other';

  // Parse UTM parameters from URL
  if ($event.page_url) {
    const url = new URL($event.page_url);
    $event.utm_source = url.searchParams.get('utm_source');
    $event.utm_medium = url.searchParams.get('utm_medium');
    $event.utm_campaign = url.searchParams.get('utm_campaign');
  }

  // Add session context
  $event.session_count = ($event.user_properties?.session_count || 0) + 1;
  $event.is_first_session = $event.session_count === 1;

  return $event;
}

4. Event Filtering​

Remove unwanted $events:

function transform($event) {
  // Filter test users
  if ($event.user_id?.startsWith('test_')) {
    return null; // Drop $event
  }

  // Filter internal IP addresses
  const internalIPs = ['192.168.', '10.0.', '172.16.'];
  if (internalIPs.some(ip => $event.context.ip?.startsWith(ip))) {
    return null;
  }

  // Filter bot traffic by user agent
  const botPatterns = /bot|crawler|spider|scraper/i;
  if (botPatterns.test($event.context.user_agent)) {
    return null;
  }

  // Filter incomplete $events
  if (!$event.user_id || !$event.timestamp) {
    return null;
  }

  return $event;
}

5. Sampling​

Reduce $event volume for high-traffic sources:

function transform($event) {
  // Sample 10% of page_view $events
  if ($event.event_name === 'page_view') {
    if (Math.random() > 0.1) {
      return null; // Drop 90% of $events
    }
    $event.is_sampled = true;
    $event.sample_rate = 0.1;
  }

  // Always keep critical events (100%)
  const criticalEvents = ['purchase_completed', 'signup_completed'];
  if (criticalEvents.includes($event.event_name)) {
    $event.is_sampled = false;
    $event.sample_rate = 1.0;
  }

  return $event;
}

6. Data Masking & Privacy​

Anonymize PII for compliance:

function transform($event) {
  // Hash email addresses
  if ($event.email) {
    $event.email_hash = hashEmail($event.email); // Built-in function
    delete $event.email; // Remove original
  }

  // Anonymize IP address (GDPR)
  if ($event.context.ip) {
    // Remove last octet: 192.168.1.100 → 192.168.1.0
    $event.context.ip = $event.context.ip.replace(/\.\d+$/, '.0');
  }

  // Pseudonymize user ID
  if ($event.user_id) {
    $event.user_id_hash = hashString($event.user_id);
    delete $event.user_id;
  }

  // Redact sensitive fields
  if ($event.credit_card) {
    $event.credit_card_last4 = $event.credit_card.slice(-4);
    delete $event.credit_card;
  }

  return $event;
}

Built-in Utility Functions​

DataBridge provides built-in utility functions for common operations:

Hashing Functions​

function transform($event) {
  // Hash email (SHA-256)
  $event.email_hash = hashEmail($event.email);
  // Output: "a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3"

  // Hash any string
  $event.user_id_hash = hashString($event.user_id);

  // MD5 hash (faster, less secure)
  $event.session_hash = md5($event.session_id);

  return $event;
}

Geo-IP Lookup​

function transform($event) {
  // Lookup geo data from IP address
  const geoData = geoIP($event.ip);

  $event.country = geoData.country; // "US"
  $event.country_name = geoData.country_name; // "United States"
  $event.region = geoData.region; // "CA"
  $event.city = geoData.city; // "San Francisco"
  $event.latitude = geoData.latitude; // 37.7749
  $event.longitude = geoData.longitude; // -122.4194
  $event.timezone = geoData.timezone; // "America/Los_Angeles"

  return $event;
}

User Agent Parsing​

function transform($event) {
  // Parse user agent string
  const ua = parseUserAgent($event.user_agent);

  $event.browser = ua.browser; // "Chrome"
  $event.browser_version = ua.browser_version; // "120.0"
  $event.os = ua.os; // "macOS"
  $event.os_version = ua.os_version; // "14.2"
  $event.device = ua.device; // "Desktop"
  $event.device_brand = ua.device_brand; // "Apple"
  $event.device_model = ua.device_model; // "MacBook Pro"

  return $event;
}

URL Parsing​

function transform($event) {
  // Parse URL components
  if ($event.page_url) {
    const url = parseURL($event.page_url);

    $event.url_protocol = url.protocol; // "https:"
    $event.url_host = url.host; // "example.com"
    $event.url_path = url.pathname; // "/products/123"
    $event.url_query = url.search; // "?utm_source=google"
    $event.url_hash = url.hash; // "#reviews"

    // Extract query parameters
    $event.utm_source = url.searchParams.get('utm_source');
    $event.utm_medium = url.searchParams.get('utm_medium');
  }

  return $event;
}

Advanced Patterns​

Error Handling​

Always handle errors gracefully:

function transform($event) {
  try {
    // Risky operation
    $event.parsed_data = JSON.parse($event.raw_json);
    $event.total = $event.parsed_data.items.reduce((sum, item) => sum + item.price, 0);
  } catch (error) {
    // Log error but don't drop $event
    console.error('Transformation error:', error.message);
    $event._transform_error = error.message;
    $event._transform_failed = true;
    // Continue with original $event
  }

  return $event;
}

Type Conversion​

Ensure correct data types:

function transform($event) {
  // String to number
  if (typeof $event.quantity === 'string') {
    $event.quantity = parseInt($event.quantity, 10);
  }

  // Number to string
  if (typeof $event.order_id === 'number') {
    $event.order_id = $event.order_id.toString();
  }

  // Boolean conversion
  $event.is_active = $event.status === 'active';
  $event.has_discount = parseFloat($event.discount || 0) > 0;

  // Array conversion
  if (typeof $event.tags === 'string') {
    $event.tags = $event.tags.split(',').map(t => t.trim());
  }

  return $event;
}

Complex Field Mapping​

Restructure nested objects:

function transform($event) {
  // Flatten nested user object
  if ($event.user) {
    $event.user_id = $event.user.id;
    $event.user_email = $event.user.email;
    $event.user_name = $event.user.name;
    $event.user_created_at = $event.user.created_at;
    delete $event.user; // Remove original nested object
  }

  // Create nested structure
  $event.purchase_info = {
    order_id: $event.order_id,
    amount: $event.amount,
    currency: $event.currency,
    items: $event.items,
    timestamp: $event.timestamp
  };

  // Clean up top-level fields
  delete $event.order_id;
  delete $event.amount;
  delete $event.currency;
  delete $event.items;

  return $event;
}

Lookup Tables​

Use constant lookups for enrichment:

function transform($event) {
  // Country code to currency mapping
  const countryCurrencyMap = {
    'US': 'USD',
    'GB': 'GBP',
    'EU': 'EUR',
    'JP': 'JPY',
    'CN': 'CNY'
  };

  if (!$event.currency && $event.country) {
    $event.currency = countryCurrencyMap[$event.country] || 'USD';
  }

  // Product category mapping
  const productCategories = {
    'PROD-001': 'Electronics',
    'PROD-002': 'Clothing',
    'PROD-003': 'Books'
  };

  if ($event.product_id) {
    $event.product_category = productCategories[$event.product_id] || 'Other';
  }

  return $event;
}

Async Functions (External API Calls)​

For external data enrichment, use async functions:

async function transform($event) {
  // Lookup user profile from API
  if ($event.user_id) {
    try {
      const userProfile = await fetch(`https://api.yourcompany.com/users/${$event.user_id}`);
      $event.user_segment = userProfile.segment;
      $event.user_lifetime_value = userProfile.ltv;
      $event.user_tier = userProfile.tier;
    } catch (error) {
      console.error('User lookup failed:', error);
      $event._enrichment_error = 'user_lookup_failed';
    }
  }

  return $event;
}

⚠️ Important notes on async functions:

  • Maximum execution time: 1 second per $event
  • Failed API calls won't drop $events
  • Use caching to reduce API calls

Testing Functions​

1. Use the Function Editor​

DataBridge provides a built-in function editor with:

  • Syntax highlighting
  • Live validation
  • Test runner with sample events

2. Test with Sample $events​

// Test input
{
  "event_name": "purchase_completed",
  "user_id": "user_12345",
  "amount": 149.99,
  "currency": "usd",
  "timestamp": "2024-01-15T10:30:00Z"
}

// After transformation
{
  "event_name": "purchase_completed",
  "user_id": "user_12345",
  "amount": 149.99,
  "currency": "USD", // normalized
  "amount_with_tax": 161.99, // calculated
  "is_high_value": false, // derived
  "event_date": "2024-01-15", // extracted
  "timestamp": "2024-01-15T10:30:00Z"
}

3. Monitor Function Performance​

Track function execution in the dashboard:

  • Execution time (avg, p50, p95, p99)
  • Error rate (% of failed executions)
  • Dropped events (filtered via return null)
  • Transformation success rate

Best Practices​

1. Keep Functions Simple​

Good:

function transform($event) {
  $event.amount_with_tax = $event.amount * 1.08;
  return $event;
}

Bad (too complex):

function transform($event) {
  // 200 lines of complex logic
  // Multiple nested conditions
  // Hard to debug
}

Solution: Split complex logic into multiple functions

2. Handle Null/Undefined Values​

function transform($event) {
  // Safe: Check existence first
  if ($event.amount) {
    $event.total = $event.amount * 1.08;
  }

  // Safe: Use optional chaining
  $event.city = $event.user?.address?.city;

  // Safe: Provide defaults
  $event.quantity = $event.quantity || 1;

  return $event;
}

3. Use Meaningful Variable Names​

Good:

const taxRate = 0.08;
const amountWithTax = $event.amount * (1 + taxRate);

Bad:

const x = 0.08;
const y = $event.amount * (1 + x);

4. Log Debug Information​

function transform($event) {
  console.log('Processing $event:', $event.event_name);
  console.log('User ID:', $event.user_id);

  if (!$event.user_id) {
    console.warn('Missing user_id for $event:', $event._databridge.$event_id);
  }

  return $event;
}

Logs appear in the function execution log dashboard.

5. Document Your Logic​

function transform($event) {
  // Enriches purchase $events with tax calculations and flags
  // - Adds amount_with_tax (8% tax rate)
  // - Adds is_high_value flag (>$1000)
  // - Normalizes currency to uppercase
    
  // Tax calculation
  const TAX_RATE = 0.08;
  $event.amount_with_tax = $event.amount * (1 + TAX_RATE);

  // High-value flag
  const HIGH_VALUE_THRESHOLD = 1000;
  $event.is_high_value = $event.amount > HIGH_VALUE_THRESHOLD;

  // Normalize currency
  $event.currency = ($event.currency || 'USD').toUpperCase();

  return $event;
}

Performance Optimization​

1. Avoid Heavy Computations​

Bad:

// Complex regex on every $event
const complexRegex = /very|complex|regex|with|many|alternatives/i;
$event.matched = complexRegex.test($event.text);

Good:

// Simple, fast checks
$event.matched = $event.text.includes('keyword');

2. Early Returns​

Exit early for filtered $events:

function transform($event) {
  // Filter first (fast)
  if ($event.user_id?.startsWith('test_')) {
    return null;
  }

  // Then do expensive operations
  $event.enriched_data = expensiveOperation($event);

  return $event;
}

Function Limits​

LimitValuePlan
Max execution time200msFree/Startup
Max execution time1sPro/Enterprise
Max function size10KBAll plans
Max console.log output10KBAll plans
Memory per execution64MBAll plans

Next Steps​

Now that you understand Functions:

  1. Create your first function in a pipeline
  2. Test with sample data in the function editor
  3. Monitor function performance in the dashboard
  4. Explore built-in enrichments for common use cases
  5. Join our community to share transformation patterns