Traffic Logging and Log Management for Tyk

Guide to configuring, managing, and analyzing API traffic logs in Tyk deployments. Learn how to set up Tyk Pump, integrate with log management systems, and gain insights from your API traffic data.

Traffic Logging and Log Management for Tyk

This guide covers strategies for configuring, managing, and analyzing API traffic logs in Tyk deployments, helping you gain insights from your API traffic while maintaining efficient log storage.

Traffic Logging Fundamentals

Understanding Tyk Logging

Tyk provides several types of logs:

• Gateway system logs: Operational logs from the Gateway itself
• API request logs: Detailed logs of API requests and responses
• Analytics records: Aggregated data about API usage
• Audit logs: Records of administrative actions

The primary mechanism for collecting and processing API traffic data is Tyk Pump, which reads analytics data from Redis and forwards it to various destinations.

Logging Use Cases

Effective traffic logging supports multiple use cases:

• Troubleshooting: Diagnose API issues and errors
• Security monitoring: Detect unusual patterns or potential attacks
• Compliance: Meet regulatory requirements for data handling
• Performance analysis: Identify slow endpoints or performance trends
• Business insights: Understand API usage patterns and user behavior

Tyk Pump Configuration

Pump Architecture

Tyk Pump:

• Reads analytics data from Redis
• Processes and formats the data
• Forwards to configured storage backends
• Manages purging of processed data

Installation and Setup

Basic steps to install and configure Tyk Pump:

1. Install Tyk Pump:

   # For Ubuntu/Debian
   sudo apt-get install tyk-pump
   

2. Configure pump.conf:

   {
     "analytics_storage_type": "redis",
     "analytics_storage_config": {
       "type": "redis",
       "host": "localhost",
       "port": 6379
     },
     "purge_delay": 10,
     "pumps": {
       "mongo": {
         "name": "mongo",
         "meta": {
           "collection_name": "tyk_analytics",
           "mongo_url": "mongodb://localhost/tyk_analytics"
         }
       }
     }
   }
   

3. Start the service:

   sudo service tyk-pump start
   

Pump Types and Configuration

Tyk Pump supports multiple destinations:

MongoDB Pump

"mongo": {
  "name": "mongo",
  "meta": {
    "collection_name": "tyk_analytics",
    "mongo_url": "mongodb://username:password@localhost:27017/tyk_analytics",
    "collection_cap_max_size_bytes": 1048576,
    "collection_cap_enable": true
  }
}

ElasticSearch Pump

"elasticsearch": {
  "name": "elasticsearch",
  "meta": {
    "index_name": "tyk_analytics",
    "elasticsearch_url": "http://localhost:9200",
    "document_type": "tyk_analytics",
    "rolling_index": false,
    "extended_stats": false,
    "version": "7"
  }
}

CSV Pump

"csv": {
  "name": "csv",
  "meta": {
    "csv_dir": "./log/",
    "flush_interval": 5
  }
}

Kafka Pump

"kafka": {
  "name": "kafka",
  "meta": {
    "broker": ["localhost:9092"],
    "topic": "tyk-analytics",
    "compressed": true
  }
}

Pump Performance Tuning

Optimize Pump performance with these settings:

{
  "analytics_storage_config": {
    "optimisation_max_idle": 100,
    "optimisation_max_active": 100
  },
  "purge_delay": 10,
  "omit_detailed_recording": false,
  "enable_aggregate_lookups": true,
  "storage_expiration_time": 60
}

Key settings to tune:

• optimisation_max_idle/active: Connection pool settings
• purge_delay: How often to purge processed data (seconds)
• omit_detailed_recording: Disable detailed logs for performance
• storage_expiration_time: How long to keep data in Redis

Log Storage Options

MongoDB Storage

MongoDB is the default storage for Tyk analytics:

• Advantages:

  • Native integration with Tyk Dashboard
  • Good for moderate volume deployments
  • Flexible querying capabilities
  • Built-in aggregation framework

• Configuration best practices:

  • Enable WiredTiger storage engine
  • Configure appropriate index strategy
  • Set up capped collections for size management
  • Implement replication for high availability

ElasticSearch Storage

ElasticSearch provides powerful search and analysis capabilities:

• Advantages:

  • Advanced search capabilities
  • Scalable for high-volume deployments
  • Strong visualization options with Kibana
  • Good for distributed environments

• Configuration best practices:

  • Implement proper index lifecycle management
  • Configure appropriate sharding
  • Set up index templates for mapping
  • Use aliases for index rotation

Cloud Logging Services

Cloud-native logging services offer managed solutions:

• AWS CloudWatch Logs
• Google Cloud Logging
• Azure Monitor Logs

Log Management

Log Rotation

Implement log rotation to manage file-based logs:

• Gateway logs: Configure logrotate for system logs
• CSV logs: Implement date-based rotation
• Database logs: Use time-based partitioning or capped collections

Example logrotate configuration for Gateway logs:

/var/log/tyk-gateway.log {
    daily
    rotate 7
    compress
    delaycompress
    postrotate
        service tyk-gateway reload > /dev/null
    endscript
}

Retention Policies

Implement appropriate retention policies:

• Short-term operational data: 7-30 days
• Medium-term analytics: 90-180 days
• Long-term business insights: 1+ years
• Compliance data: Based on regulatory requirements

Configure MongoDB TTL index for automatic expiration:

db.tyk_analytics.createIndex( { "timestamp": 1 }, { expireAfterSeconds: 7776000 } ) // 90 days

Sensitive Data Handling

PII Identification

Identify sensitive data in logs:

• Personal identifiers (names, emails, IPs)
• Financial information
• Authentication credentials
• Session identifiers
• Health information

Data Masking

Implement data masking for sensitive information:

{
  "analytics_config": {
    "type": "redis",
    "sanitize_field_names": [
      "authorization",
      "password",
      "credit_card",
      "card_number"
    ]
  }
}

Compliance Requirements

Address compliance requirements:

• GDPR: Implement data minimization and retention limits
• HIPAA: Ensure PHI protection and access controls
• PCI DSS: Mask card data and implement strict access controls
• SOC 2: Maintain audit trails of log access

Log Analysis and Visualization

Basic Log Analysis

Implement basic log analysis capabilities:

• Search and filtering by API, response code, time
• Pattern matching for error identification
• Trend analysis for performance and usage
• Correlation between errors and system events

ELK Stack Integration

Integrate with ELK Stack for advanced analysis:

1. Configure Logstash to process Tyk logs
2. Create Kibana dashboards for visualization:
   • API usage by endpoint
   • Error rates and patterns
   • Geographic distribution of requests
   • Response time trends

Implementation Example: Financial Services API Logging

This example demonstrates a comprehensive logging implementation for a financial services company with strict compliance requirements.

Requirements:

• Detailed logging for compliance and security
• 7-year retention of transaction logs
• PII protection and masking
• Real-time security monitoring
• Performance analysis capabilities

Implementation:

1. Pump Configuration:

   {
     "analytics_storage_type": "redis",
     "analytics_storage_config": {
       "type": "redis",
       "host": "redis-master",
       "port": 6379,
       "optimisation_max_idle": 200,
       "optimisation_max_active": 200
     },
     "purge_delay": 5,
     "pumps": {
       "elasticsearch": {
         "name": "elasticsearch",
         "meta": {
           "index_name": "tyk_analytics",
           "elasticsearch_url": "https://elasticsearch:9200",
           "rolling_index": true,
           "extended_stats": true,
           "sensitive_fields": ["authorization", "token", "key", "api_key", "password"],
           "sensitive_fields_replacement": "****"
         }
       },
       "mongo-storage": {
         "name": "mongo",
         "meta": {
           "collection_name": "tyk_analytics",
           "mongo_url": "mongodb://mongo-analytics:27017/tyk_analytics"
         }
       },
       "mongo-archive": {
         "name": "mongo",
         "meta": {
           "collection_name": "tyk_analytics_archive",
           "mongo_url": "mongodb://mongo-archive:27017/tyk_analytics_archive"
         }
       }
     }
   }
   

2. Retention Strategy:

   • ElasticSearch: 90-day retention with index lifecycle management
   • Primary MongoDB: 1-year retention with time-based partitioning
   • Archive MongoDB: 7-year retention with cold storage integration

3. Results:

   • Comprehensive compliance with financial regulations
   • 99.9% log capture rate
   • Effective PII protection with field-level masking
   • 65% reduction in storage costs through tiered storage
   • Real-time security monitoring with automated alerts

Best Practices

Configuration Best Practices

• Start with MongoDB pump for basic deployments
• Add ElasticSearch for advanced search and analysis
• Use multiple pumps for different retention needs
• Configure appropriate batch sizes and intervals
• Regularly verify log delivery and storage

Storage Best Practices

• Implement tiered storage for cost optimization
• Set appropriate retention periods by data type
• Use compression for long-term storage
• Implement backup strategies for log data
• Monitor storage growth and plan capacity

Security Best Practices

• Mask sensitive data at collection time
• Encrypt logs in transit and at rest
• Implement strict access controls
• Maintain audit trails of log access
• Regularly review logging for compliance

Next Steps

• Monitoring and Alerting
• Security Hardening
• Disaster Recovery