Commit 3be236bc authored by Cameron McFarland's avatar Cameron McFarland

Adding more details pertaining to the zip file, and also setting up TF to...

Adding more details pertaining to the zip file, and also setting up TF to rebuild the zip file if it changes.
parent 0b667610
......@@ -30,6 +30,20 @@ Kinesis is how SnowPlow hands off data from collector to enricher to s3loader.
Kinesis Firehose takes events from a stream and applies a Lambda function
to the event, then write it into the S3 bucket.
## Lambda Function
Firehose uses a Lambda function to format events written to S3. The Lambda
function code is in the file ```lambda/```. As of this
writing, all this function does is append a newline to the end of each event
before it is written to S3.
The AWS provider for Terraform requires a zip file of this code to update or
create the Lambda function. There is a data archive object in the config with
the name ```snowplow_lambda_event_formatter_archive``` that build the zip file
from the function python script. If the hash of that file changes, terraform
will try to update the function. It's possible that the zip file hash chances,
but no code changes were made. This is ok to replace on the fly in the Lambda
## DynamoDB
The enricher and s3loader nodes use DynamoDB to track Kinesis state. Normally
these tables would be allocated by Terraform, but if the nodes themselves don't
......@@ -66,6 +66,12 @@ data "template_file" "snowplow_s3_bucket_policy" {
template = "${file("${path.module}/templates/gitlab-com-snowplow-events.policy.json")}"
data "archive_file" "snowplow_lambda_event_formatter_archive" {
type = "zip"
source_file = "${path.module}/lambda/"
output_path = "${path.module}/lambda/"
// Policies
resource "aws_iam_policy" "snowplow_collector_policy" {
description = "Policy the allows the collector to access other AWS services such as Kinesis."
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment