Commit 09055b22 authored by Cameron McFarland's avatar Cameron McFarland

Updating some meta information.

parent 1a075635
......@@ -4,7 +4,8 @@ This configuration uses the following AWS services to host SnowPlow. There
may be more in use, but these are the primary services.
1. EC2 (Auto Scaling Groups, Launch Configurations, ELB, Target Groups,
Security Groups)
1. Kinesis
1. Kinesis (Streams and Firehose)
1. Lambda
1. DynamoDB
1. IAM (Policies and Roles)
1. S3
......@@ -15,9 +16,8 @@ If you want to know more about the SnowPlow infrastructure, please consult
the design document. XXX
## SnowPlow Installs and Configs
There are three types of SnowPlow nodes (Collectors, Enrichers, and S3Loaders)
and they are all configured and installed via user-data in the launch
configurations.
There are two types of SnowPlow nodes (Collectors and Enrichers) and they are
all configured and installed via user-data in the launch configurations.
## Kinesis Streams
Kinesis is how SnowPlow hands off data from collector to enricher to s3loader.
......@@ -25,7 +25,10 @@ Kinesis is how SnowPlow hands off data from collector to enricher to s3loader.
* snowplow-raw-bad
* snowplow-enriched-good
* snowplow-enriched-bad
* snowplow-s3loader-bad
## Kinesis Firehose and Lambda
Kinesis Firehose takes events from a stream and applies a Lambda function
to the event, then write it into the S3 bucket.
## DynamoDB
The enricher and s3loader nodes use DynamoDB to track Kinesis state. Normally
......@@ -37,11 +40,6 @@ do that on their own.
* SnowplowEnrich-gitlab-us-east-1
* SnowplowS3Loader-gitlab-us-east-1
## Firehose
The data format conversion sections of the Firehose deliver configs are
placeholder configs since data format conversion is disabled. Terraform still
seems to want to configure everything if it's disabled.
## Launch Config Changes and Production Instances
Updating the launch config will apply to new systems coming up in the
auto-scaling group. But existing EC2 instances won't be changed. You will
......@@ -67,6 +65,5 @@ S3Loader:
"I realize now my folly: the app name needs to be different between the enricher and loader ergo the 2 dynamoDB tables were conflicting. Everything makes so much sense now…"
Last steps:
Need to get firehose provisioned. It's weird.
Are we using the right SSH key?
Did we clean up everything we made for testing?
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment