Metadata ingestion setup for AWS S3

In order for Model-Prime to process your robologs, you must grant the ingest role read access to the S3 bucket in which they are stored.

This guide requires no prerequisite S3 knowledge. If you already know how to configure AWS, or do so using cloudformation/terraform/AWS CLI, you may skip to the Grant Model-Prime Permission to the Bucket section, which will provide you with the IAM information to use with your existing workflow. Otherwise, this section will follow and extend the steps from the official AWS guide.

Step 1. Create a bucket, if necessary

If you already have a bucket which contains your robolog data, simply take note of the region it is in, then feel free to skip this step.

Create an S3 bucket in which you intend to store the robologs to be ingested by Model-Prime. You may do so by following the official S3 bucket creation user guide.

When selecting the region, note that Model-Prime currently supports the following regions:

• us-east-1
• us-east-2

If you require support for a different region, send a request to support@model-prime.com, and the Model-Prime support team will enable ingestion for your selected region.

It is best to not modify the default configuration of this newly created bucket to ensure that your sensitive information is NOT made accessible to the public. As an additional security step, we advise that any S3 bucket shared with us should ONLY contain files or objects that you and your company are willing to share with Model-Prime.

Step 2. Grant Model-Prime permission to the bucket

Model-Prime needs the s3:GetObject permission on your bucket to ingest the data it contains. This permission may be granted within the bucket's Permissions tab as shown below.

In order to provide you with a thorough example, we have created a bucket named modelprime.example.bucket and will grant it the necessary permissions for ingestion.

Once in this tab, scroll down to the bucket policy widget and press Edit.

Once editing, add a policy similar to the following. Replace modelprime.example.bucket with the name of your bucket.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<provided-Model-Prime-account-number>:role/ingest"
      },
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::modelprime.example.bucket/*"]
    }
  ]
}

Take note that we have set an Allow for Model-Prime's ingest role to perform the s3:GetObject action on the example bucket. You must use the same Principal value as shown above. The Resource value should refer to your bucket ARN which may be conveniently copied from the AWS web console.

Encrypted buckets

If you've elected to encrypt your bucket, you will need to grant Model-Prime's IAM ingest role permission to use the bucket's KMS key. Official instructions on creating a cross-account KMS key before associating it with your bucket may be found here.

Following the official cross-account key creation instructions will result in a policy that provides access to Model-Prime's root account. You will need to edit the key policy using the editor provided during the key creation Review step, or after creating the key, by changing all references of <provided-Model-Prime-account-number>:root to <provided-Model-Prime-account-number>:role/ingest.

If you've already created the key, follow these steps to edit the key policy:

1. Go to the AWS Key Management Service dashboard.
2. Select Customer managed keys.
3. Click on the key alias for the key associated with your ingest bucket.
4. Scroll down to the Key policy section and click the Edit button.
5. Ensure that the following policy statements are included:.

{
  "Sid": "Allow use of the key",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::<provided-Model-Prime-account-number>:role/ingest"
  },
  "Action": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey*",
    "kms:DescribeKey"
  ],
  "Resource": "*"
}

{
  "Sid": "Allow attachment of persistent resources",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::<provided-Model-Prime-account-number>:role/ingest"
  },
  "Action": ["kms:CreateGrant", "kms:ListGrants", "kms:RevokeGrant"],
  "Resource": "*",
  "Condition": {
    "Bool": {
      "kms:GrantIsForAWSResource": "true"
    }
  }
}

Next steps

Model-Prime will now have the ability to read objects from your bucket. In order to receive information about what exists in the bucket, ingestion must be invoked. Proceed to the next section to learn more about the options available for triggering ingest.