TL;DR
The series.
The Problem with EC2-Based Splunk Integration
The AWS Blog's architecture works, but it comes with operational overhead:
| Concern | EC2-Based | Serverless |
|---|---|---|
| Monthly cost | ~$66 fixed | ~$6 pay-per-use |
| OS patching | Monthly | None |
| Agent updates | Manual (UF + syslog-ng) | None |
| Scaling | Manual instance resize | Automatic (Lambda concurrency) |
| Availability | Single AZ (unless you add redundancy) | Multi-AZ by default |
| Time to deploy | Hours (provision + configure) | 30 minutes (CloudFormation) |
If you're already running this EC2 pattern and want to modernize, this article shows you how — with a parallel deployment strategy that ensures zero data loss during cutover.
Architecture
CODE
┌──────────────────────────────────────────────────────────┐
│ FSx for ONTAP │
│ │
│ Audit Volume ──→ S3 Access Point │
│ │ │
│ ▼ │
│ EventBridge Scheduler (rate: 5 min) │
│ │ │
│ ▼ │
│ Lambda (Python 3.12) │
│ • Reads audit logs via S3 AP │
│ • Parses JSON/EVTX │
│ • Formats as Splunk HEC events │
│ • Sends with Authorization: Splunk <token> │
│ • Checkpoints in SSM Parameter Store │
│ │ │
│ ▼ │
│ Splunk HEC │
│ https://<splunk>:8088/services/collector/event │
│ Response: {"text":"Success","code":0} │
│ │
│ SPL: index=fsxn_audit sourcetype=fsxn:ontap:audit │
└──────────────────────────────────────────────────────────┘
High-Volume Alternative: Firehose Path
For sustained >1000 events/sec, use Kinesis Data Firehose with its built-in Splunk destination:
CODE
FSx for ONTAP → S3 AP → Lambda (transform) → Kinesis Data Firehose → Splunk HEC
A separate template-firehose.yaml is provided for this path.
Migration Strategy (Zero Data Loss)
Phase 1: Parallel Deployment (Day 1-3)
Deploy the serverless stack alongside the existing EC2 pipeline. Use a separate Splunk index for validation:
CODE
aws cloudformation deploy \
--template-file integrations/splunk-serverless/template.yaml \
--stack-name fsxn-splunk-integration \
--parameter-overrides \
S3AccessPointArn=<S3_AP_ARN> \
SplunkHecTokenSecretArn=<SECRET_ARN> \
SplunkHecEndpoint=https://splunk.example.com:8088 \
S3BucketName=<BUCKET> \
SplunkIndex=fsxn_audit_serverless \
--capabilities CAPABILITY_IAM
Compare events between old and new pipelines for 48 hours:
CODE
| stats count by index
| where index IN ("fsxn_audit", "fsxn_audit_serverless")
Phase 2: Cutover (Day 4-5)
Once event parity is confirmed:
- Update the stack to use the production index (
fsxn_audit) - Stop the syslog-ng and UF services on EC2 (don't terminate yet)
- Monitor for 24 hours
Phase 3: Cleanup (Day 7+)
CODE
# Terminate EC2 instances
# Remove security groups, IAM roles, EBS volumes
# Delete old CloudFormation/Terraform resources
What Changes for Splunk Users
Unchanged ✅
- Index name and sourcetype (configurable)
- SPL queries — same field names
- Dashboards and saved searches
- Alert rules
Changed ⚠️
hostfield: EC2 hostname → SVM name
sourcefield: syslog path →fsxn-observability
- Delivery latency: near-real-time (syslog) → polling interval (default 5 min)
HEC Event Format
CODE
{
"time": 1716508800,
"host": "svm-prod-01",
"source": "fsxn-observability",
"sourcetype": "fsxn:ontap:audit",
"index": "fsxn_audit",
"event": {
"event_type": "4663",
"user": "[email protected]",
"operation": "ReadData",
"path": "/vol/data/report.pdf",
"result": "Success",
"client_ip": "10.0.1.50"
}
}
SPL Query Examples
CODE
# Failed access attempts
index=fsxn_audit sourcetype=fsxn:ontap:audit result=Failure
| stats count by user, path
| sort -count
# Operations timeline
index=fsxn_audit sourcetype=fsxn:ontap:audit
| timechart span=5m count by operation
# Top users
index=fsxn_audit sourcetype=fsxn:ontap:audit
| stats count by user
| sort -count
| head 20
# Specific user investigation
index=fsxn_audit sourcetype=fsxn:ontap:audit user="[email protected]"
| table _time, operation, path, result, client_ip
Cost Comparison
| Component | EC2-Based (monthly) | Serverless (monthly) | Savings |
|---|---|---|---|
| EC2 instances (2× t3.medium) | $60 | $0 | 100% |
| EBS volumes (2× 20GB) | $6 | $0 | 100% |
| Lambda | $0 | ~$5 | — |
| EventBridge Scheduler | $0 | ~$0.01 | — |
| Secrets Manager | $0 | ~$0.40 | — |
| Total | $66 | $6 | 91% |
Note: EC2 cost assumes 2× t3.medium (as per the (
AuthenticationorChangedata model) for compatibility with Splunk Enterprise Security correlation searches
Index pre-creation: Thefsxn_auditindex must be created before first ingestion (Splunk Cloud: Admin Console; Enterprise:indexes.conf)
EMS webhooks: Real-time ARP ransomware detection alerts
FPolicy: Sub-second file operation streaming
Production Readiness: Progress from Level 1 (this Quick Start) to Level 4 (Enterprise) — see the :
Level
What You Get
Go/No-Go to Next
Level 1 (this Quick Start)
Audit poller + DLQ
Logs arrive, checkpoint advances, DLQ empty 24h
Level 2
+ Splunk dashboards + alerts
SLOs met 7 days, security review done
Level 3
+ DynamoDB ledger + poison-pill
SLOs met 30 days, compliance pack
Level 4
+ OTel Collector + redaction
Multi-backend, PII redaction, DR tested
Data classification: Splunk receives
userandpathfields (PII/sensitive). For Splunk Cloud, data is processed in the vendor's infrastructure. For self-hosted Splunk Enterprise, data stays in your VPC. See |
Part 2:
Part 4:
Part 6:
Part 8: EC2 to Serverless: Modernizing Splunk Integration (this post)
Part 9:
Part 11:
Part 13:↗ Original-Artikel auf dev.to lesenVollständiger Original-BerichtAusführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.
SOCIAL SHARE CARD GENERATOR