Post

AWS - boto3 - EBS

Posted
By Grace L
15 min read

EBS

  • EBS
    • Client
    • Paginators

Client

class EBS.Client

  • A low-level client representing Amazon Elastic Block Store (EBS)
  • use the Amazon Elastic Block Store (Amazon EBS) direct APIs to create EBS snapshots, write data directly to snapshots, read data on snapshots, and identify the differences or changes between two snapshots.
  • If you’re an independent software vendor (ISV) who offers backup services for Amazon EBS, the EBS direct APIs make it more efficient and cost-effective to track incremental changes on EBS volumes through snapshots. This can be done without having to create new volumes from snapshots, and then use Amazon Elastic Compute Cloud (Amazon EC2) instances to compare the differences.
  • create incremental snapshots directly from data on-premises into EBS volumes and the cloud to use for quick disaster recovery.
  • With the ability to write and read snapshots, write on-premises data to an EBS snapshot during a disaster. Then after recovery, restore it back to AWS or on-premises from the snapshot. no longer need to build and maintain complex mechanisms to copy data to and from Amazon EBS.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82

import boto3

ebsclient = boto3.client('ebs')

can_paginate()

# complete_snapshot()
# Seals and completes the snapshot
response = ebsclient.complete_snapshot(
    SnapshotId='string',
    ChangedBlocksCount=123,
    Checksum='string',
    ChecksumAlgorithm='SHA256',
    ChecksumAggregationMethod='LINEAR'
)

generate_presigned_url()


get_paginator()

# get_snapshot_block()
# Returns the data in a block in an Amazon Elastic Block Store snapshot.
response = ebsclient.get_snapshot_block(
    SnapshotId='string',
    BlockIndex=123,
    BlockToken='string'
)


get_waiter()

# list_changed_blocks()
# Returns information about the blocks that are different between two Amazon Elastic Block Store snapshots of the same volume/snapshot lineage.
response = ebsclient.list_changed_blocks(
    FirstSnapshotId='string',
    SecondSnapshotId='string',
    NextToken='string',
    MaxResults=123,
    StartingBlockIndex=123
)

# list_snapshot_blocks()
# Returns information about the blocks in an Amazon Elastic Block Store snapshot.
response = ebsclient.list_snapshot_blocks(
    SnapshotId='string',
    NextToken='string',
    MaxResults=123,
    StartingBlockIndex=123
)

# put_snapshot_block()
# Writes a block of data to a snapshot.
response = ebsclient.put_snapshot_block(
    SnapshotId='string',
    BlockIndex=123,
    BlockData=b'bytes'|file,
    DataLength=123,
    Progress=123,
    Checksum='string',
    ChecksumAlgorithm='SHA256'
)


# start_snapshot()
# Creates a new Amazon EBS snapshot.
response = ebsclient.start_snapshot(
    VolumeSize=123,
    ParentSnapshotId='string',
    Tags=[
        {
            'Key': 'string',
            'Value': 'string'
        },
    ],
    Description='string',
    ClientToken='string',
    Encrypted=True|False,
    KmsKeyArn='string',
    Timeout=123
)

can_paginate(operation_name)

Check if an operation can be paginated.

Parameters

  • operation_name (string): is the same name as the method name on the client.
  • For example, if the method name is create_foo, and you’d normally invoke the operation as client.create_foo(kwargs_), if the create_foo operation can be paginated, you can use the call client.get_paginator(“create_foo”).

Returns

  • True if the operation can be paginated,
  • False otherwise.

complete_snapshot(kwargs_)

Seals and completes the snapshot after all of the required blocks of data have been written to it.

  • Completing the snapshot changes the status to completed .
  • cannot write new blocks to a snapshot after it has been completed.

Request Syntax

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

response = ebsclient.complete_snapshot(
    SnapshotId='string',
    # [REQUIRED]: The ID of the snapshot.
    ChangedBlocksCount=123,
    # [REQUIRED]: The number of blocks that were written to the snapshot.
    Checksum='string',
    # An aggregated Base-64 SHA256 checksum based on the checksums of each written block.
    # To generate the aggregated checksum using the linear aggregation method,
    # arrange the checksums for each written block in ascending order of their block index,
    # concatenate them to form a single string,
    # and then generate the checksum on the entire string using the SHA256 algorithm.
    ChecksumAlgorithm='SHA256',
    # The algorithm used to generate the checksum.
    # Currently, the only supported algorithm is SHA256 .
    ChecksumAggregationMethod='LINEAR'
    # The aggregation method used to generate the checksum.
    # Currently, the only supported aggregation method is LINEAR .
)

Returns

  • Return type: dict
  • Response Syntax:
1

{'Status': 'completed'|'pending'|'error'}   # The status of the snapshot.

Exceptions

1
2
3
4
5
6

* EBS.Client.exceptions.AccessDeniedException
* EBS.Client.exceptions.ValidationException
* EBS.Client.exceptions.ResourceNotFoundException
* EBS.Client.exceptions.RequestThrottledException
* EBS.Client.exceptions.ServiceQuotaExceededException
* EBS.Client.exceptions.InternalServerException

generate_presigned_url(ClientMethod, Params=None, ExpiresIn=3600, HttpMethod=None)

Generate a presigned url given a client, its method, and arguments

Parameters

  • ClientMethod (string) – The client method to presign for
  • Params (dict) – The parameters normally passed to ClientMethod.
  • ExpiresIn (int) – The number of seconds the presigned url is valid for. By default it expires in an hour (3600 seconds)
  • HttpMethod (string) – The http method to use on the generated url. By default, the http method is whatever is used in the method’s model.

Returns: The presigned url

get_paginator(operation_name)

Create a paginator for an operation.

Parameters

  • operation_name (string), is the same name as the method name on the client. For example,
  • if the method name is create_foo, invoke the operation as client.create_foo(**kwargs),
  • if the create_foo operation can be paginated, you can use the call client.get_paginator("create_foo").

Raises OperationNotPageableError

  • Raised if the operation is not pageable.
  • use the client.can_paginate method to check if an operation is pageable.

Returns

  • Return type: L{botocore.paginate.Paginator}
  • Returns: A paginator object

get_snapshot_block(kwargs_)

Returns the data in a block in an Amazon Elastic Block Store snapshot.

API:

1
2
3
4
5
6
7
8
9
10
11

# GetSnapshotBlock

# Request Syntax
GET /snapshots/snapshotId/blocks/blockIndex?blockToken=BlockToken HTTP/1.1

# Response Syntax
HTTP/1.1 200
x-amz-Data-Length: DataLength
x-amz-Checksum: Checksum
x-amz-Checksum-Algorithm: ChecksumAlgorithm
BlockData

Request Syntax

1
2
3
4
5
6
7
8
9
10

response = client.get_snapshot_block(
    SnapshotId='string',
    # The ID of the snapshot containing the block
    BlockIndex=123,
    # The block index of the block from which to get data.
    # Obtain the BlockIndex by running the ListChangedBlocks or ListSnapshotBlocks.
    BlockToken='string'
    # The block token of the block from which to get data.
    # Obtain the BlockToken by running the ListChangedBlocks or ListSnapshotBlocks operations.
)

Returns

  • Return type: dict
  • Returns: Response Syntax
    1
2
3
4
5
6
7
8
9
10

    {
  'DataLength': 123,
  # The size of the data in the block.
  'BlockData': StreamingBody(),
  # The data content of the block.
  'Checksum': 'string',
  # The checksum generated for the block, Base64 encoded.
  'ChecksumAlgorithm': 'SHA256'
  # The algorithm used to generate the checksum for the block, such as SHA256.
}

Exceptions

  • EBS.Client.exceptions.AccessDeniedException
  • EBS.Client.exceptions.ValidationException
  • EBS.Client.exceptions.ResourceNotFoundException
  • EBS.Client.exceptions.RequestThrottledException
  • EBS.Client.exceptions.ServiceQuotaExceededException
  • EBS.Client.exceptions.InternalServerException

get_waiter(waiter_name)

Returns an object that can wait for some condition.

Parameters

  • waiter_name (str) – The name of the waiter to get.

Returns

  • The specified waiter object.
  • Return type: botocore.waiter.Waiter

list_changed_blocks(kwargs_)

Returns information about the blocks that are different between two Amazon Elastic Block Store snapshots of the same volume/snapshot lineage.

Request Syntax

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

response = client.list_changed_blocks(
    FirstSnapshotId='string',
    # The ID of the first snapshot to use for the comparison.
    # Warning: The FirstSnapshotID parameter must be specified with a SecondSnapshotId parameter; otherwise, an error occurs.
    SecondSnapshotId='string',
    # **[REQUIRED]**
    # The ID of the second snapshot to use for the comparison.
    # Warning: The SecondSnapshotId parameter must be specified with a FirstSnapshotID parameter; otherwise, an error occurs.
    NextToken='string',
    # The token to request the next page of results.
    MaxResults=123,
    # The number of results to return.
    StartingBlockIndex=123
    # The block index from which the comparison should start.
    # The list in the response will start from this block index or the next valid block index in the snapshots.
)

Returns

  • Return type: dict
  • Response Syntax
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

# _(dict) --_
{
    'ChangedBlocks':
    # An array of objects containing information about the changed blocks.
    [
    # A block of data in an Amazon Elastic Block Store snapshot that is different from another snapshot of the same volume/snapshot lineage.
        {
            'BlockIndex': 123,
            # The block index.
            'FirstBlockToken': 'string',
            # The block token for the block index of the FirstSnapshotId specified in the ListChangedBlocks operation. This value is absent if the first snapshot does not have the changed block that is on the second snapshot.
            'SecondBlockToken': 'string'
            # The block token for the block index of the SecondSnapshotId specified in the ListChangedBlocks operation.
        },
    ],
    'ExpiryTime': datetime(2015, 1, 1),
    # The time when the BlockToken expires.
    'VolumeSize': 123,
    'BlockSize': 123,
    'NextToken': 'string'
    # The token to use to retrieve the next page of results. This value is null when there are no more results to return.
}

Exceptions

  • EBS.Client.exceptions.AccessDeniedException
  • EBS.Client.exceptions.ValidationException
  • EBS.Client.exceptions.ResourceNotFoundException
  • EBS.Client.exceptions.RequestThrottledException
  • EBS.Client.exceptions.ServiceQuotaExceededException
  • EBS.Client.exceptions.InternalServerException

list_snapshot_blocks(kwargs_)

Returns information about the blocks in an Amazon Elastic Block Store snapshot.

See also: AWS API Documentation

Request Syntax

1
2
3
4
5
6

response = client.list_snapshot_blocks(
    SnapshotId='string',
    NextToken='string',
    MaxResults=123,
    StartingBlockIndex=123
)

Parameters

  • SnapshotId (string) –
    • [REQUIRED]
    • The ID of the snapshot from which to get block indexes and block tokens.
  • NextToken (string) –
    • The token to request the next page of results.
  • MaxResults (integer) –
    • The number of results to return.
  • StartingBlockIndex (integer) –
    • The block index from which the list should start. The list in the response will start from this block index or the next valid block index in the snapshot.

Returns:

  • Return type: dict
  • Response Syntax
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

{
    # An array of objects containing information about the blocks.
    'Blocks': [
        # A block of data in an Amazon Elastic Block Store snapshot.
        {
            'BlockIndex': 123,
            'BlockToken': 'string'
        },
    ],
    'ExpiryTime': datetime(2015, 1, 1),
    'VolumeSize': 123,
    'BlockSize': 123,
    # The token to use to retrieve the next page of results. This value is null when there are no more results to return.
    'NextToken': 'string'
}

Exceptions

  • EBS.Client.exceptions.AccessDeniedException
  • EBS.Client.exceptions.ValidationException
  • EBS.Client.exceptions.ResourceNotFoundException
  • EBS.Client.exceptions.RequestThrottledException
  • EBS.Client.exceptions.ServiceQuotaExceededException
  • EBS.Client.exceptions.InternalServerException

put_snapshot_block(kwargs_)

  • Writes a block of data to a snapshot.
  • If the specified block contains data, the existing data is overwritten.
  • The target snapshot must be in the pending state.

Data written to a snapshot must be aligned with 512-byte sectors.

Request Syntax

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

response = client.put_snapshot_block(
    SnapshotId='string',
    BlockIndex=123,
    # The block index of the block in which to write the data.
    # A block index is a logical index in units of 512 KiB blocks.
    # To identify the block index, divide the logical offset of the data in the logical volume by the block size (logical offset of data/524288 ).
    # The logical offset of the data must be 512 KiB aligned.
    BlockData=b'bytes'|file,
    # The data to write to the block.
    # The block data is not signed as part of the Signature Version 4 signing process.
    # As a result, you must generate and provide a Base64-encoded SHA256 checksum for the block data using the **x-amz-Checksum** header.
    # Also, must specify the checksum algorithm using the **x-amz-Checksum-Algorithm** header.
    # The checksum that you provide is part of the Signature Version 4 signing process.
    # It is validated against a checksum generated by Amazon EBS to ensure the validity and authenticity of the data.
    # If the checksums do not correspond, the request fails.
    DataLength=524288,
    # The size of the data to write to the block, in bytes.
    # Currently, the only supported size is 524288 .
    Progress=123,
    Checksum='string',
    # A Base64-encoded SHA256 checksum of the data. Only SHA256 checksums are supported.
    ChecksumAlgorithm='SHA256'
    # The algorithm used to generate the checksum. Currently, the only supported algorithm is SHA256 .
)

Returns

  • Return type: dict
  • Response Syntax
1
2
3
4
5
6

{
    'Checksum': 'string',
    # The SHA256 checksum generated for the block data by Amazon EBS.
    'ChecksumAlgorithm': 'SHA256'
    # The algorithm used by Amazon EBS to generate the checksum.
}

Exceptions

  • EBS.Client.exceptions.AccessDeniedException
  • EBS.Client.exceptions.ValidationException
  • EBS.Client.exceptions.ResourceNotFoundException
  • EBS.Client.exceptions.RequestThrottledException
  • EBS.Client.exceptions.ServiceQuotaExceededException
  • EBS.Client.exceptions.InternalServerException

start_snapshot(kwargs_)

  • Creates a new Amazon EBS snapshot.
  • The new snapshot enters the pending state after the request completes.
  • After creating the snapshot, use PutSnapshotBlock to write blocks of data to the snapshot.

Request Syntax

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

response = client.start_snapshot(
    VolumeSize=123,
    # The size of the volume, in GiB. The maximum size is 16384 GiB (16 TiB).
    ParentSnapshotId='string',
    # The ID of the parent snapshot.
    # If there is no parent snapshot, or creating the first snapshot for an on-premises volume, omit this parameter.
    # If account is enabled for encryption by default, you cannot use an unencrypted snapshot as a parent snapshot. first create an encrypted copy of the parent snapshot using CopySnapshot
    Tags=[
        {
            'Key': 'string',
            'Value': 'string'
        },
    ],
    Description='string',
    ClientToken='string',
    # A unique, case-sensitive identifier you provide to ensure the idempotency of the request.
    # Idempotency ensures that an API request completes only once.
    # With an idempotent request, if the original request completes successfully. The subsequent retries with the same client token return the result from the original successful request and they have no additional effect.
    # autopopulated if not provided. If you do not specify a client token, one is automatically generated by the AWS SDK.
    Encrypted=True|False,
    # Indicates whether to encrypt the snapshot.
    # To create an encrypted snapshot = true.
    # To create an unencrypted snapshot, omit this parameter.
    # If you specify a value for **ParentSnapshotId** , omit this parameter.
    # If true, the snapshot is encrypted using the CMK specified using the **KmsKeyArn** parameter.
    # If no value is specified for **KmsKeyArn** , the default CMK for account is used. If no default CMK has been specified for account, the AWS managed CMK is used. To set a default CMK for account, use ModifyEbsDefaultKmsKeyId
    # If account is enabled for encryption by default, you cannot set this parameter to false . In this case, you can omit this parameter.
    KmsKeyArn='string',
    # The Amazon Resource Name (ARN) of the AWS Key Management Service (AWS KMS) customer master key (CMK) to be used to encrypt the snapshot.
    # If you do not specify a CMK, the default AWS managed CMK is used.
    # If you specify a **ParentSnapshotId** , omit this parameter; the snapshot will be encrypted using the same CMK that was used to encrypt the parent snapshot.
    # If **Encrypted** is set to true , you must specify a CMK ARN.
    Timeout=123
    # The amount of time (minutes) after which the snapshot is automatically cancelled if:
    # - No blocks are written to the snapshot.
    # - The snapshot is not completed after writing the last block of data.
    # If no value is specified, the timeout defaults to 60 minutes.
)

Return

  • Return type: dict
  • Response Syntax
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

{
    'Description': 'string',
    # The description of the snapshot.
    'SnapshotId': 'string',
    'OwnerId': 'string',
    'Status': 'completed'|'pending'|'error',
    'StartTime': datetime(2015, 1, 1),
    # The timestamp when the snapshot was created.
    'VolumeSize': 123,
    # The size of the volume, in GiB.
    'BlockSize': 123,
    'Tags': [
        {
            'Key': 'string',
            'Value': 'string'
        },
    ],
    'ParentSnapshotId': 'string',
    'KmsKeyArn': 'string'
    # The Amazon Resource Name (ARN) of the AWS Key Management Service (AWS KMS) customer master key (CMK) used to encrypt the snapshot.
}

Exceptions

  • EBS.Client.exceptions.AccessDeniedException
  • EBS.Client.exceptions.ValidationException
  • EBS.Client.exceptions.RequestThrottledException
  • EBS.Client.exceptions.ResourceNotFoundException
  • EBS.Client.exceptions.ServiceQuotaExceededException
  • EBS.Client.exceptions.InternalServerException
  • EBS.Client.exceptions.ConcurrentLimitExceededException
  • EBS.Client.exceptions.ConflictException

example

Enable EBS Default Encryption

  1. do via AWS CLI
1

$ aws ec2 enable-ebs-encryption-by-default
  1. script for one region
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

import boto3

region = 'eu-west-1'
ec2connection = boto3.client('ec2', region)
print ("Checking AWS Region: " + region)

status = ec2connection.get_ebs_encryption_by_default()  # the ebs current status

print (" ------------- unit test: if it is already encrypted ------------- ")
result = status["EbsEncryptionByDefault"]
if result == True:
    print ("Activated, nothing to do")
else:
    print("Not activated, activation in progress")
    response = ec2connection.enable_ebs_encryption_by_default()
    print("Setup default EBS Encryption for region", region,": ",  response['EbsEncryptionByDefault'])
  1. script for all the region
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

import boto3

# --------------------- check for all the region
# AWS_REGION = 'eu-west-1'
# session = boto3.Session(region_name=AWS_REGION)
# ec2 = session.client('ec2')

ec2 = boto3.Session(region_name='eu-west-1').client('ec2')

def main(event, context):
    ec2_regions = [region['RegionName'] for region in ec2.describe_regions()['Regions']]
    # For all AWS Regions
    for region in ec2_regions:
        conn = boto3.client('ec2', region_name=region)
        print ("Checking AWS Region: " + region)
        status = conn.get_ebs_encryption_by_default()
        print ("===="*10)
        result = status["EbsEncryptionByDefault"]
        if result == True:
            print ("Activated, nothing to do")
        else:
            print("Not activated, activation in progress")
            conn.enable_ebs_encryption_by_default()

if __name__ == '__main__':
    main(0,0)

.

01AWS, boto3
AWS
This post is licensed under CC BY 4.0 by the author.

Comments powered by Disqus.