Cyral
Get Started Sign In

Protect S3 Data Endpoints with Cyral

You can protect your S3 buckets using Cyral. See Track an S3 storage location for set-up instructions. 

The article below is a supplement to the main Cyral documentation and explains advanced configuration settings.

Track the S3 endpoint in Cyral

  • In the Cyral management console, navigate to the Data Repos tab and click the plus button.

  • In the pop-up dialog, enter Amazon S3 as the Repository Type, and click Track.


Install the sidecar

  • If you don't already have a sidecar deployed that can serve this repository, add it now as shown in Install a sidecar.


Associate the S3 endpoint with your sidecar

To protect a repository and allow users to connect to it, you must associate the tracked repository with its sidecar.

  • In the Sidecars tab, select the sidecar to which you'd like to assign the repository and click the plus sign.

  • In the Assign a Repository window, choose the name of the S3 repository you created above, and specify the hostname and port where data users will connect to this repository. The TLS toggle is always set to ON for S3 repositories.

  • In the pop-up dialog, select your repository and click Track.

Your S3 data endpoint is now accessible through the Cyral sidecar. See the next section for connection instructions.




Connect to an S3 data endpoint through Cyral


These instructions assume you already have performed the following tasks:

  • Tracked the S3 endpoint in Cyral

  • Deployed a sidecar

  • Associated the S3 endpoint with the sidecar in the Cyral Control Plane


For connecting to your S3 endpoint through the sidecar, we need to take the following steps:


Get the sidecar endpoint

Go to the Cyral Control Plane and find the sidecar endpoint associated with the S3 data repository. To do this, navigate to the Sidecars page, click the name of the sidecar, and note the Sidecar Endpoint for 



In the above figure, the host is www.sidecar-endpoint.com and the port is 453.

Set the sidecar endpoint as the HTTP proxy address

In a shell session, we need to export the following environment variables:



and if the host machine is a AWS EC2 instance, you also need to export the variable:


export NO_PROXY=169.254.169.254


Detailed information and settings for configuring the proxy endpoint for different systems is available at: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-proxy.html


If you're using Nifi to access S3, see Using NiFi to access data in a Cyral-protected S3 bucket.


Get an SSO token and add it to your credentials file

Append the following profile information to the AWS credentials file. This file is usually found at ~/.aws/credentials .


[sidecar]

aws_access_key_id=userA01@company.com:c17c7b65c159544f75bd98e91d584

aws_secret_access_key=none


  • userA01@company.com  is the email address used to log in to the Control Plane.

  • c17c7b65c159544f75bd98e91d584 is the access token copied from the Control Plane. You can also get the token from the Cyral CLI token retriever for SSO, gimme_db_token. For instructions, see Connect to S3 data using the CLI token retriever.


Access S3 using the profile you created

Run S3 commands specifying the previously created sidecar profile:


> aws s3 ls --profile sidecar


This uses the credentials collected from the Control Plane and validates the user, sending all traffic through the sidecar. Logs should be available at the log location configured in your Cyral installation, such as Cloudwatch, ELK, or Splunk.



Alternative and extra configurations

Adding a certificate authority bundle to the CLI tool

The sidecar needs to intercept the TLS communication between the user and AWS servers. For keeping the connection secure, the sidecar signs the messages using its own certificate. Some CLI clients and tools may complain about the certificate used by the sidecar, so an extra configuration needs to be performed:


First, download the certificate bundle from the control plane using the following command:

curl https://<CONTROLPLANE_ENDPOINT>:8000/v1/templates/ca_bundle -o cyral_ca_bundle.pem


Next, provide the CA bundle to the AWS CLI using one of the following commands.

  1. Using an environment variable:

export AWS_CA_BUNDLE=/path/to/cyral_ca_bundle.pem


  • cyral_ca_bundle.pem: This file is used by the AWS CLI tool to validate the certificate sent by the sidecar. This file will be shared by the Cyral’s Support team.

  1. Using profile configuration: Append the following information to the configuration file (the location will be either or defined by the environment variable)

[profile sidecar]

ca_bundle = /path/to/cyral_ca_bundle.pem


Both options above (a) and (b) are equivalent and only one needs to be performed


Using a profile configuration instead of environment variables for proxy settings


Throughout this guide, we've used environment variables to configure the proxy settings used by the AWS CLI tool. An alternative approach is to add the proxy settings to the configuration profile instead. To do this, we need to add a third-party plugin, "awscli-plugin-s3-proxy", to the AWS CLI tool.


For the steps below, we are assuming the profile sidecar_sample_profile will be used.


Note: If your system doesn't recognize the python -m pipcommand, you can try one of the following:
  • Replace python -m pip with just pip in the CLI commands
  • Replace python -m pip with either python -m pip3 or just pip3 (for environments with pip support  for Python 2 and Python 3)
  • Install pip following: https://pip.pypa.io/en/stable/installing/


Procedure


1. Install the plugin, "awscli-plugin-s3-proxy", using python pip:

python -m pip install awscli-plugin-s3-proxy --user


2. If you are using the AWS CLI v2 (aws --version), you will need to define the appropriate location for the plugin:

a. Get the plugin location:

python -m pip show awscli-plugin-s3-proxy

It is expected to have a Location key in the generated output, for example:

  • Linux: Location: /home/<username>/.local/lib/python3.8/site-packages
  • Mac: Location: /Users/<username>/.local/lib/python3.8/site-packages
  • Windows: Location: c:\Users\<username>\Application Data\python\python38\site-packages

b. Set the plugin location with the command:

aws configure set plugins.cli_legacy_plugin_path <LOCATION>

which for our Mac output above, would be, for example:

aws configure set plugins.cli_legacy_plugin_path /Users/<username>/.local/lib/python3.8/site-packages


3. Configure using the AWS CLI:

aws configure set plugins.s3-proxy awscli_plugin_s3_proxy


4. Add the sidecar as a proxy for S3:

aws configure --profile sidecar_sample_profile set s3.proxy http://<YOUR_SIDECAR_ADDRESS>:453

replacing <YOUR_SIDECAR_ADDRESS> with the actual address.


5. Add the Cyral CA bundle to the same profile:

aws configure --profile sidecar_sample_profile set ca_bundle /path/to/cyral_ca_bundle.pem


Enable SSO login on your S3 data endpoint

To enable SSO login on your S3 data repository, select the S3 repository and go to the Advanced tab. Here, make the following settings.

  1. Choose your Identity Provider from the drop-down box.

  2. Unselect the checkbox Allow native authentication.

  3. Under Enforcement, enable Block on violations.




Provide the IAM roles needed for accessing S3

Under “Local Accounts'' tab for the S3 repo specify the IAM roles that provide various levels of access to S3. Depending on which end user is connecting, the sidecar will assume one of these roles and make the S3 request using the local account assigned to that user. For more information on IAM role settings see "Make AWS IAM role settings” at the end of this section


  1. In the Cyral control plane UI, go to Repositories, click Local Accounts, and click Track Account.

  2. In the Track Account form, enter the IAM role ARN

  3. Click Track.


In the below screenshot a role providing full s3 access is shown as example


Map an SSO user or group to the IAM Role

When a user authenticates, they can be mapped to use a specific IAM role to access S3 based on their username, or based on their membership in an SSO group. Set up the mapping as follows.


  1. In the Repositories page, click Identity to Account Map and click the plus sign.

  2. Choose User or Group as the identity type.

  3. In the Identity field, specify the SSO user name or group name as it's written in your identity service.

  4. In the Local Account field, choose the name of the IAM role

  5. In the Duration field, set a length of validity for the access, or click Unlimited to grant access that will not expire automatically.

  6. Click Create.


In the below screenshot users belonging to “Data Analysts” SSO group are mapped to user “SidecarS3FullAccess” role 



Make AWS IAM role settings

It is important to make sure that the IAM role associated with the sidecar is trusted by the IAM roles used for managing S3. 


Find your sidecar host role

After deploying the sidecar, the IAM role associated with the sidecar is created with name <sidecar-cft-stack-name>SidecarHostRole-*

Below is an example sidecar host role created for a sidecar with stack name “jc-t01”


> aws iam list-roles



// Sidecar Role

{

    "Path": "/",

    "RoleName": "jc-t01-SidecarHostRole-MOVF2C5ORWCY",

    "RoleId": "AROA2TGP77HETTOXZZB46",

    "Arn": "arn:aws:iam::<accountId>:role/jc-t01-SidecarHostRole-MOVF2C5ORWCY",

    "CreateDate": "2020-12-22T23:58:42Z",

    "AssumeRolePolicyDocument": {

        "Version": "2012-10-17",

        "Statement": [

            {

                "Effect": "Allow",

                "Principal": {

                    "Service": "ec2.amazonaws.com"

                },

                "Action": "sts:AssumeRole"

            }

        ]

    },

    "Description": "",

    "MaxSessionDuration": 3600

},


Create suitable IAM roles for S3 access and establish trust with sidecar role

The various roles and the extent of permissions the roles get is up to the organization. In the below example, we have show two roles “SidecarReadOnlyRole” and “SidecarS3FullAccess”. There is a trust relationship between these roles and the SidcarHostRole, which allows sidecar to assume these roles.


// S3 Access Read Only

{

    "Path": "/",

    "RoleName": "SidecarReadOnlyRole",

    "RoleId": "AROA2TGP77HEZGE2FX3G4",

    "Arn": "arn:aws:iam::<accountId>:role/SidecarReadOnlyRole",

    "CreateDate": "2020-12-14T18:48:08Z",

    "AssumeRolePolicyDocument": {

        "Version": "2012-10-17",

        "Statement": [

            {

                "Effect": "Allow",

                "Principal": {

                    "Service": "ec2.amazonaws.com"

                },

                "Action": "sts:AssumeRole"

            },

            {

                "Effect": "Allow",

                "Principal": {

                    // ARN of the sidecar IAM role

                    "AWS":"arn:aws:iam::<accountid>:role/jc-t01-SidecarHostRole-MOVF2C5ORWCY"

                },

                "Action": "sts:AssumeRole"

            }

        ]

    },

    "Description": "Allows EC2 instances to call AWS services on your behalf.",

    "MaxSessionDuration": 3600

}




// S3 Role Full Access

{

    "Path": "/",

    "RoleName": "SidecarS3FullAccess",

    "RoleId": "AROA2TGP77HEYZQT3DH4A",

    "Arn": "arn:aws:iam::<accountId>:role/SidecarS3FullAccess",

    "CreateDate": "2020-12-14T18:48:32Z",

    "AssumeRolePolicyDocument": {

        "Version": "2012-10-17",

        "Statement": [

            {

                "Effect": "Allow",

                "Principal": {

                    "Service": "ec2.amazonaws.com"

                },

                "Action": "sts:AssumeRole"

            },

            {

                "Effect": "Allow",

                "Principal": {

                    // ARN of the sidecar IAM role

                    "AWS": "arn:aws:iam::<accountId>:role/jc-t01-SidecarHostRole-MOVF2C5ORWCY"

                },

                "Action": "sts:AssumeRole"

            }

        ]

    },

    "Description": "Allows EC2 instances to call AWS services on your behalf.",

    "MaxSessionDuration": 3600

}



By having the sidecar role as a trusted entity for your S3 management roles, you are allowing the sidecar to assume these custom roles, when required, for connecting to the S3 servers when handling SSO connections.


This information can also be edited in the AWS Console in the IAM → Roles section.



Users can connect to S3 using the CLI token retriever for SSO

Data users can use the Cyral CLI token retriever for SSO, gimme_db_token, to quickly authenticate and connect to their S3 data endpoints. For instructions, see Connect to S3 data using the CLI token retriever.




Did you find it helpful? Yes No

Send feedback
Sorry we couldn't be helpful. Help us improve this article with your feedback.