Cross-region data transfer from NFS server to Amazon EFS file system using AWS DataSync

Overview

This AWS DataSync solution transfers data from an EC2-hosted NFS share in VPC A (us-east-1) to an Amazon EFS file system in VPC B (us-west-2). The two VPCs are connected through a peering connection. This setup simulates a typical on-premises to cloud data migration scenario.

An NFS server uses the NFS protocol to share files and directories over a network.

Amazon EFS is a fully managed, serverless cloud file storage service that provides elastic, shared file storage for Linux-based workloads and applications running on AWS services like EC2.

AWS DataSync, a fully managed, high-speed data transfer service, communicates with both storage locations via VPC endpoints, utilizing a private, fast and secure connection to accomplish the transfer.

VPC Endpoints provide private interface entry points to AWS services from within a VPC. While many AWS services have public endpoints that can be accessed over the internet, it is a best practice to use VPC endpoints to communicate with AWS services securely from within a VPC.

Architecture Diagram

 to enable  and attach an EC2 role to the NFS server for session manager to work. Attach the following AWS managed policies to the EC2 role:

1. AmazonSSMManagedEC2InstanceDefaultPolicy


2. AmazonSSMManagedInstanceCore

.

.

#!/bin/bash

set -e

sleep 180

# Check if the secondary drive exists

if ! lsblk /dev/xvdb &>/dev/null; then

    echo “/dev/xvdb not found. Exiting.”

    exit 1

fi



mkfs.ext4 /dev/xvdb

mkdir -p /mnt/data

mount /dev/xvdb /mnt/data

echo ‘/dev/xvdb /mnt/data ext4 defaults,nofail 0 2’ | sudo tee -a /etc/fstab



# Create and set up NFS share directory

TARGET_DIR=”/mnt/data/nfs_share”

PREFIX=”test”

EXT=”.txt”

mkdir -p $TARGET_DIR

chown -R nobody:nobody $TARGET_DIR

chmod 777 $TARGET_DIR



# Function to generate files

generate_files() {

local start=$1

local end=$2

local size_range=$3

local block_size=$4

local group_label=$5



local start_time=$(date +%s)

echo “Starting generation of $group_label at $(date)”



for i in $(seq -f “%04g” $start $end); do

    FILE=”${TARGET_DIR}/${PREFIX}${i}${EXT}”

    SIZE=$((RANDOM % size_range + 1))

    dd if=/dev/urandom of=”$FILE” bs=$block_size count=$SIZE status=none &

done

wait



local end_time=$(date +%s)

echo “Finished generation of $group_label at $(date)”

local elapsed_time=$((end_time - start_time))

echo “Time taken for $group_label: ${elapsed_time} seconds”

}



# Generate file groups with time tracking

generate_files 1 1000 100 1K “1000 files (1KB to 100KB)” &

generate_files 1001 1100 100 1M “100 files (1MB to 100MB)” &



wait

echo “File generation complete in $TARGET_DIR”



sed -i “\|^$TARGET_DIR |d” /etc/exports

echo “$TARGET_DIR *(rw,sync,no_root_squash,no_all_squash)” | sudo tee -a /etc/exports > /dev/null

systemctl start nfs-server

systemctl enable nfs-server

exportfs -rav

Verify that you can connect to the NFS server via Session Manager.

US-WEST-2

VPC B

1. Create a target VPC in us-west-2 with 172.31.0.0/16 Cidr.


2. Create two public subnets and an internet gateway.


3. Create two private subnets and a NAT gateway.


4. Follow the same steps as in Region A to enable Session Manager to access the NFS client.


5. 
   

   Create and attach an SSM Endpoint Security Group to all the endpoints:

   
   
   
   
   
   1. Inbound: 
      
      
      
      
      1. HTTPS from DataSync agent.
      
      
      2. NFS from VPC B.
      
      
      3. 1024 – 1064 from DataSync agent.
      
      
      
      
      
      
   
   
   
   2. Outbound
      
      
      
      
      1. All traffic to VPC B.
      
      
      
      
      
      
   
   
   
   
   
   




6. Attach the same EC2 role created from VPC A to the NFS client if you are in the same account. For different accounts, you need to create a new EC2 role for the NFS client.




7. 
   
   

   .



8. 
   t3.micro instance type.


9. Proceed without a key pair.


10. PrivateSubnetA.


11. Auto-assign public IP disabled.
    


12. Create a DataSync agent security group as follows:
    
    
    
    
    1. Inbound: 
       
       
       
       
       1. HTTPS from VPC B.
       
       
       2. HTTP from NFS client.
       
       
       3. ICMP from NFS client.
       
       
       
       
       
       
    
    
    
    2. Outbound
       
       
       
       
       1. All traffic to VPC A and B.
       
       
       
       
       
       
    
    
    
    
    
    



13. 
    
    20 GB gp3 EBS volume.

VPC B Endpoints

Follow similar procedures from US-EAST-1 to create VPC endpoints for:

1. SSM.


2. EC2-Messages.


3. SSM-Messages.


4. EFS.


5. DataSync.

.

EFS file system

1. Create an EFS Security Group with the following rules:
   
   
   
   
   1. Inbound: 
      
      
      
      
      1. NFS traffic from VPC B.
      
      
      2. NFS traffic from NFS Client.
      
      
      3. NFS traffic from DataSync agent. 
      
      
      4. NFS traffic from SSM Endpoint.
      
      
      5. HTTPS from VPC B and SSM Endpoint.
      
      
      6. ICMP from NFS client.
      
      
      
      
      
      
   
   
   
   2. Outbound
      
      
      
      
      1. All traffic to VPC B.
      
      
      
      
      
      
   
   
   
   
   
   



2. 
   
   .


3. Optional name.


4. VPC ID (Requester): VPC A ID.
   


5. Select another VPC to peer with: My account.
   


6. Region: Another Region.
   


7. VPC ID (Accepter): VPC B ID.
   

, select the peering connection available.

1. Update the private route table for VPC B to forward Region A traffic to the peering connection.

DataSync agent

Obtain the activation code of the DataSync agent from the NFS client.

# Test connectivity to the agent

nc -vz 172.31.x.x 80 



# Obtain activation code

sudo curl “http://<datasync-agent-ip>/?gatewayType=SYNC&activationRegion=us-west-2&privateLinkEndpoint=<datasync-vpce-ip>&endpointType=PRIVATE_LINK&no_redirect”



# NB: Replace <datasync-agent-ip> and <datasync-vpce-ip> with actual values

Activate the DataSync agent using the activation key.

aws datasync create-agent \

    --agent-name “datasync-agent” \

    --activation-key “xxxxx-xxxxx-xxxxx-xxxxx-xxxxx” \

    --vpc-endpoint-id “vpce-xxxxxxxxxxxxxxxxx” \

    --subnet-arns “arn:aws:ec2:us-west-2:accountId:subnet/subnet-xxxxxxxxxxx” \

    --security-group-arns “arn:aws:ec2:us-west-2:accountId:security-group/sg-xxxxxxxxxxxx”

Create the NFS location.

aws datasync create-location-nfs \

    --server-hostname “10.0.x.x” \

    --subdirectory “/mnt/data/nfs_share” \

    --on-prem-config AgentArns=”arn:aws:datasync:us-west-2:accountId:agent/agent-xxxxxxxxxxx”

Create the EFS location.

aws datasync create-location-efs \

    --efs-filesystem-arn “arn:aws:elasticfilesystem:us-west-2:accountId:file-system/fs-xxxxxxxxxxxxx” \

    --ec2-config SubnetArn=”arn:aws:ec2:us-west-2:accountId:subnet/subnet-xxxxxxxxxxxxx”,SecurityGroupArns=”arn:aws:ec2:us-west-2:accountId:security-group/sg-xxxxxxxxxxxxxxx”

Create a task that connects both locations and start the task.

aws datasync create-task \

    --source-location-arn “arn:aws:datasync:us-west-2:accountId:location/loc-xxxxxxxxxxxxxx” \

    --destination-location-arn “arn:aws:datasync:us-west-2:accountId:location/loc-xxxxxxxxxxxxxx”

.