Skip to main content

Data Transformation & Enrichment

Data transformation pipeline

Turn Raw Events Into Analytics-Ready Data​

Process, enrich and transform your data in real-time as it flows through the pipeline. Shape events before they reach your warehouse instead of cleaning them up later.

Real-Time Data Transformation​

Process Data In-Flight​

Transform data as it streams through DataBridge, before it's written to your destination. This approach reduces downstream processing, lowers storage costs and ensures your data is immediately ready for analysis.

Benefits:

  • Immediate Value: Data arrives at your warehouse ready for analysis
  • Reduced Costs: Pay only for delivered events after filtering and deduplication
  • Lower Latency: No need for batch jobs to clean and transform data
  • Single Pipeline: One transformation layer instead of multiple ETL jobs

Transformation Types​

Field Mapping & Renaming​

Standardize field names across different data sources:

// Before transformation
{
  "usr_id": "123",
  "mail": "user@example.com"
}

// After transformation
{
  "user_id": "123",
  "email": "user@example.com"
}

Type Conversion​

Convert data types to match your schema requirements:

// Convert strings to appropriate types
{
  "price": "99.99",      // String
  "quantity": "5",       // String
  "timestamp": "1640000000"  // Unix timestamp as string
}

// Becomes
{
  "price": 99.99,        // Number
  "quantity": 5,         // Integer
  "timestamp": "2021-12-20T13:46:40Z"  // ISO datetime
}

Field Extraction​

Extract nested values or parse complex strings:

// Extract from nested objects
{
  "user": {
    "profile": {
      "email": "user@example.com",
      "name": "John Doe"
    }
  }
}

// Flatten to
{
  "user_email": "user@example.com",
  "user_name": "John Doe"
}

Data Masking & PII Redaction​

Protect sensitive information:

// Before masking
{
  "email": "john.doe@example.com",
  "ssn": "123-45-6789",
  "credit_card": "4111111111111111"
}

// After masking
{
  "email": "j***@example.com",
  "ssn": "***-**-6789",
  "credit_card": "************1111"
}

Conditional Transformations​

Apply transformations based on conditions:

// Add VIP flag based on purchase amount
if ($event.purchase_amount > 1000) {
  $event.customer_tier = "VIP";
} else if ($event.purchase_amount > 500) {
  $event.customer_tier = "Premium";
} else {
  $event.customer_tier = "Standard";
}

Support for External Lookups​

Enrich events with data from your own APIs or third-party services:

// Look up company info from IP address
const res = await fetch(
  `https://api.example.com/enrich/company?ip=${$event.ip_address}`,
  { headers: { "Authorization": "Bearer ${SECRET.ENRICH_API_KEY}" } }
);
const company = await res.json();

$event.company_name = company.name;
$event.company_industry = company.industry;
$event.company_size = company.employee_count;

Data Enrichment​

Pre-Built Enrichment Functions​

Enhance your events with contextual data using DataBridge's library of pre-built enrichment functions:

Geolocation Enrichment​

Add geographic context from IP addresses:

// Input
{
  "user_id": "123",
  "ip_address": "8.8.8.8"
}

// Enriched output
{
  "user_id": "123",
  "ip_address": "8.8.8.8",
  "geo": {
    "country": "United States",
    "country_code": "US",
    "region": "California",
    "city": "Mountain View",
    "postal_code": "94043",
    "latitude": 37.386,
    "longitude": -122.084,
    "timezone": "America/Los_Angeles"
  }
}

User Agent Parsing​

Extract device and browser information:

// Input
{
  "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0 like Mac OS X)..."
}

// Enriched output
{
  "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0 like Mac OS X)...",
  "device": {
    "type": "mobile",
    "brand": "Apple",
    "model": "iPhone",
    "os": "iOS",
    "os_version": "14.0",
    "browser": "Safari",
    "browser_version": "14.0"
  }
}

Referrer Parsing​

Understand traffic sources:

// Input
{
  "referrer": "https://www.google.com/search?q=data+quality"
}

// Enriched output
{
  "referrer": "https://www.google.com/search?q=data+quality",
  "referrer_parsed": {
    "source": "google",
    "medium": "organic",
    "campaign": null,
    "term": "data quality",
    "type": "search"
  }
}

UTM Parameter Extraction​

Parse marketing campaign parameters:

// Input
{
  "url": "https://example.com?utm_source=facebook&utm_medium=cpc&utm_campaign=summer_sale"
}

// Enriched output
{
  "url": "https://example.com?utm_source=facebook&utm_medium=cpc&utm_campaign=summer_sale",
  "campaign": {
    "source": "facebook",
    "medium": "cpc",
    "campaign": "summer_sale",
    "content": null,
    "term": null
  }
}

Timestamp Normalization​

Convert various timestamp formats to ISO 8601:

// Accepts multiple formats
"2024-01-15"                    // Date only
"1705334400"                    // Unix timestamp (seconds)
"1705334400000"                 // Unix timestamp (milliseconds)
"2024-01-15T10:00:00-08:00"    // ISO with timezone

// Always outputs
"2024-01-15T18:00:00.000Z"     // ISO 8601 UTC

Currency Conversion​

Add converted amounts for international transactions:

// Input
{
  "amount": 100,
  "currency": "EUR"
}

// Enriched with current exchange rates
{
  "amount": 100,
  "currency": "EUR",
  "amount_usd": 109.50,
  "exchange_rate": 1.095,
  "exchange_rate_date": "2024-01-15"
}

Custom Functions​

Build Your Own Transformations​

For business-specific logic, create custom transformation functions using JavaScript:

Function Example:

// Custom function to calculate customer lifetime value
function enrichWithLTV($event) {
  const purchases = $event.total_purchases || 0;
  const avgOrderValue = $event.avg_order_value || 0;
  const customerAge = $event.customer_age_days || 0;

  // Calculate projected LTV
  const purchaseFrequency = purchases / (customerAge / 30);
  const projectedLTV = purchaseFrequency * avgOrderValue * 36; // 3-year projection

  $event.customer_ltv = Math.round(projectedLTV * 100) / 100;
  $event.customer_segment =
    projectedLTV > 10000 ? "high_value" :
    projectedLTV > 5000 ? "medium_value" : "low_value";

  return $event;
}

Use Cases for Custom Functions:

  • Calculate derived metrics (conversion rate, engagement score, etc.)
  • Implement business rules and logic
  • Integrate with external APIs for data lookup
  • Apply complex conditional transformations
  • Aggregate or roll up data points

Function Management​

Testing & Debugging

  • Test functions with sample events
  • View transformation results in real-time
  • Debug with detailed error logs
  • Monitor function performance
  • Timeout protection (max 2 seconds per event)

Flexible Processing Pipeline​

Filter & Deduplicate​

Event Filtering​

Remove unwanted events before they're stored:

// Filter out bot traffic
if ($event.user_agent.includes('bot') || $event.user_agent.includes('crawler')) {
  return null; // Event will be discarded
}

// Filter internal traffic
if ($event.ip_address.startsWith('10.') ||
    $event.email.endsWith('@yourcompany.com')) {
  return null;
}

Common Filtering Use Cases:

  • Remove bot and crawler traffic
  • Filter out internal team events
  • Exclude test events
  • Remove events based on feature flags
  • Filter low-value events to reduce costs

Sampling​

Reduce event volume for high-frequency, low-value events:

// Sample 10% of page_view events
if ($event.event_name === 'page_view' && Math.random() > 0.1) {
  return null;
}

// Sample based on user_id for consistent sampling
const userId = $event.user_id;
const hash = simpleHash(userId);
if (hash % 10 !== 0) {  // Keep 10% of users consistently
  return null;
}

Data Validation & Correction​

Autocorrect common data quality issues:

// Trim whitespace
$event.email = $event.email?.trim();

// Standardize formats
$event.country_code = $event.country_code?.toUpperCase();
$event.email = $event.email?.toLowerCase();

// Set defaults for missing values
$event.currency = $event.currency || 'USD';
$event.quantity = $event.quantity || 1;

// Clamp values to valid ranges
$event.rating = Math.max(1, Math.min(5, $event.rating));

Performance & Scalability​

High-Performance Processing​

  • Sub-Millisecond Latency: Most transformations complete in < 10ms
  • Parallel Execution: Process multiple events simultaneously
  • Auto-Scaling: Handles traffic spikes automatically

Cost Optimization​

Pay for Delivered Events Only pay for events that pass validation and filtering. If you filter out 50% of incoming events, you only pay for the 50% that are delivered.

Example Cost Savings:

Incoming Events:     1,000,000/month
Bot Traffic:         -300,000 (filtered)
Test Events:         -50,000 (filtered)

Delivered Events:    650,000/month
Cost Savings:        35% reduction

Ready to Get Started?​

Start with our free tier (100K events/month) or explore paid plans starting at $79/month.