Commit 067236f2 authored by Cameron McFarland's avatar Cameron McFarland

Updating notes.

parent 996fac24
......@@ -2,15 +2,57 @@
Locally pull down the docker images we think we'll need.
* docker pull snowplow-docker-registry.bintray.io/snowplow/scala-stream-collector-kinesis:0.15.0
* docker pull snowplow-docker-registry.bintray.io/snowplow/stream-enrich-kinesis:0.21:0
* docker pull snowplow-docker-registry.bintray.io/snowplow/s3-loader:0.6.0
If we are going to user docker images, we'll need to make our own images since
we need to put our config in there. There just isn't a supported method to
tunnel configs in via environment variables (at least not easily).
Need to tag them and put them into the two ECR repositories.
Need to build a VPC for this entire thing. 10.32.0.0/16
There is also a subnet.
There is also a subnet. Probably should make three, in different regions.
Also need an internet gateway? And routing tables?
ECS Tasks need a IAM role to allow access to things?
Need roles/policies to allow proper access to the collectors, enrichers and s3 loaders.
ECS Tasks need a IAM role to allow access to things? Yes.
Going to probably need an elastic IP for the snowplow endpoint.
Kinesis Streams:
snowplow-raw-good
snowplow-raw-bad
snowplow-enriched-good
snowplow-enriched-bad
snowplow-s3loader-bad
DynamoDBs:
These tables will be made by the enricher and s3loader respectively. With proper roles/policies in place, there is no need to make these with TF.
SnowplowEnrich-gitlab-us-east-1
SnowplowS3Loader-gitlab-us-east-1
Volumes and config?
One way is to include the config into the docker image. Not easy to update configs.
Mount an EBS volume? Not sure how this works yet.
The default config (of which there isn't one in the pre-built images) does not support env variables.
Switch to EC2 and a dynamic LB? Why? Why not?
I switched to EC2, I made the snowplow operator user and policy and groups.
I also created an EC2 role to allow EC2 to access kinesis. This seems to be working. At least I can see a config working quickly.
Collector:
I made two kinesis streams with 1 shard each to start with. snowplow-good and snowplow-bad
OMG it worked.
[ec2-user@ip-10-32-0-206 ~]$ curl http://localhost:8000/health
OK[ec2-user@ip-10-32-0-206 ~]$
Enricher:
Need an iglu resolver config. Is it ok to use the public repo? Need to figure this out.
Also might need a dynamoDB to store stream state. The enricher will make it's own table. This is actually required since hand-making it seems to not work.
Enricher is working? OMG.
Debugging: -Dorg.slf4j.simpleLogger.defaultLogLevel=debug
Testing an event: curl http://34.227.92.217:8000/i\?e\=pv
SwLoader:
"I realize now my folly: the app name needs to be different between the enricher and loader ergo the 2 dynamoDB tables were conflicting. Everything makes so much sense now…"
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment