In the previous article of the series, we discussed in brief all the three ways. Now in this one, we will see the first way in a bit more detail.

In the technology value stream, the work flows from Development to Operations or from Dev to Ops in short. The faster the flow is, the faster the delivery of value to the customers.

How we do the above is through making work visible, reducing batch sizes, by building quality in, preventing defects from being passed to downstream work centers

The by products of the doing are

• reducing the lead time to fulfill internal and external customer requests

• increasing the quality of work

• making the organization more more responsive to the customer and market needs

Make our Work Visible

A major difference between technology and manufacturing value streams is that our work is invisible as the transfer of work between work centers can be done with a click of button

Due to the ease in doing the task, work can bounce between teams due to incomplete information or the problems remain completely invisible resulting in delay in the delivery of the promised improvements or features.

To avoid the issues mentioned above, we should make the work visible using work boards like Kanban boards. It helps in making the work visible and also in managing the work in order to make it flow from left to right as quickly as possible. Also it helps to zero in on the unnecessary hand-offs which introduce unnecessary delays. Work is considered to be done when those features are available in production

Limit Work in Process

In Manufacturing, daily work is dictated by a production schedule which is in turn governed by other factors like order due dates, parts availability etc. But in technology, the work is more dynamic

Disruptions in the manufacturing space is highly visible and costly but same in the technology is invisible.

In order to avoid that we can impose WIP limits for each column as shown earlier. It helps to see problems that prevent completion of work. The focus shifts to completing the already started work.

We will see the other principles of flow in future posts.

Buckets aren’t encrypted. But Objects are. Each object could be using different encryption settings.

There are two main types of encryption at rest:

• Client side encryption

• Server side encryption

Both of them refer to encryption at rest. Only data is encrypted but not object metadata.

Encryption at rest - Method of encryption when the objects are persistently stored on disk.

The objects being uploaded are encrypted by the client before they ever leave.The data is in cipher-text form the entire time it is stored in the bucket.

With server side encryption, the data is encrypted with https while in transit.The data is encrypted when it reaches the endpoint

With client side encryption, everything is under the user’s control.

3 types of server side encryption and is a trade off between trust, overhead, cost , resource consumption etc.

1. Server side encryption with Customer-Provided Keys (SSE-C)

2. Server side encryption with Amazon S3 managed keys (SSE-S3)

3. Server side encryption with KMS keys stored in AWS-KMS (SSE-KMS)

Two components to SSE

• The actual encryption and decryption process itself – cryptographic operations

• Generation and management of cryptographic keys.

In SSE-C, customer is responsible for the encryption keys used in the encrypt/decrypt operations

So the first major difference between client side encryption and SSE-C is that S3 service is handling the cryptographic operations. The cpu requirements are being offloaded to S3 but the customer is required to manage the keys

So the object (obj.) to be encrypted and the key is supplied to S3. Also a hash of the key is created and attached to the obj and key is discarded. This hash can’t be used to generate a new key.

If key is provided during decrypt. the hash will identify whether the same key was used to encrypt the obj.

So as the key is not present now, S3 is requested for the obj to be decrypted and the key to decrypt it.

S3 will determine the key is correct, then use it to decrypt the obj. and discard the key.

Customer needs to manage the keys, but retains the control of the cryptographic. ops. and also save cpu requirement.

In the next blog post we will see the rest of the types of server side encryption.

AWS Key Management Service (AWS KMS) lets us create, manage, and control cryptographic keys across your applications and more than 100 AWS services.

AWS KMS is a secure and resilient service that uses hardware security modules that have been validated under FIPS 140-2, or are in the process of being validated, to protect your keys.

Any AWS service using encryption uses KMS.It is a regional and public service. Every region is isolated when using KMS. It helps to create, store and manage cryptographic keys. These keys can be used to create plain text information to cipher text information and vice-versa. It is able to handle both symmetric and asymmetric keys (private asymmetric and public asymmetric keys)

It is capable of performing actual cryptographic operations. (encrypt, decrypt etc.)

The KMS keys never leave the KMS service. Can create, import, manage keys etc. But keys are locked inside KMS

It provides FIPS 140-2 compliant service which is a US security standard. Some features are compliant with level three standard. The keys were earlier known as CMK but currently they are known as KMS keys. They Are used by KMS within cryptographic operations

Users can use them, apps can use them and other AWS services can use them. KMS Keys are logical.

They can be thought of as containers for physical key material

KMS Keys contains:

• key ID: unique identifier for the key

• creation date

• key policy (a type of resource policy)

• description and state of key (active or inactive)

Every KMS Key is supported by physical key material. It is this material that is used to encrypt and decrypt things. Physical material can be generated by KMS or imported into KMS. This physical material inside KMS Keys are used to encrypt or decrypt data that’s upto 4kB in size. It has other features to overcome the 4 kB limitation.

Lets assume that there is a user who interacts with KMS. The first interaction after picking a region is creating a key. So user runs createkey operation to create KMS key. Then KMS key contains physical backing key material. This is the most important thing that KMS creates, stores and manages. KMS keys are encrypted before they are store persistently on disk. Now there is an encrypted KMS key

Next interaction might be to encrypt some data. The encrypt call is used providing the key to use and some data to encrypt. Provided that user has required permissions to access the key, KMS decrypts key & uses it to encrypt the plain-text data and returns the encrypted data to the user.

So KMS is handling this cryptographic operation

If the data needs to be decrypted, then user uses Decrypt operation and provides the encrypted data. The key to be used to decrypt data is encoded in the cipher text of the data to be decrypted. If user has decrypt permissions, KMS decrypts the KMS keys and uses it to decrypt the cipher text and provides the data in plain-text

The permissions to generate keys, encrypt and decrypt are all different. The permissions are granular Note that the KMS key does not leave KMS at any stage or stored in plain-text at any stage. To perform each operation like creating the keys, assigning the permissions, for encrypt and decrypt operations, separate permissions are required

Different users can be given different permissions w.r.t KMS. (a.k.a role separation), For example, some people might have permission to encrypt but not generate keys and assign permissions and decrypt.

Data Encryption Keys

KMS can only perform cryptographic operations only on data that has upto 4kb in size. Then how is the 4kb limitation is overcome?

KMS has another type of keys – data encryption keys (DEK)– created using KMS keys Using GenerateDataKey operation and KMS Key, a DEK is generated and this DEK can be used to encrypt data that is greater than 4kb size. DEK is linked to a specific KMS Key that was used in creating them. So KMS knows which DEK is created using which KMS key.

The DEK is not stored in KMS. It is provided to user or service using KMS and then discards it. Because KMS itself does not encrypt/decrypt using DEKs. That is done either by the user or the service that uses the DEK

Data is encrypted using plaintext DEK and then it is discarded. Generally the encrypted data and the encrypted DEK are stored together. KMS by itself doesn’t encrypt the data greater than 4 kb. Either the user or the service using the DEK does it. KMS doesn’t track usage of DEK

The same DEK can be used to encrypt multiple files or new DEKs can be generated for each of them The data and the DEK used to encrypt it are stored on same disk in an encrypted form. So the administration is easy and security is maintained

Data encrypted by DEK can be decrypted by passing the encrypted DEK back to KMS Then use the decrypted DEK is used to decrypt the data and the decrypted DEK is discarded.

Key concepts:

KMS key are stored within KMS specific to that region and never leave the region or the service

KMS Keys cannot be extracted and any interactions with KMS keys is done through APIs provided by KMS.

They are either AWS Owned or Customer owned

AWS Owned KMS keys are owned and managed by AWS services to be used in multiple accounts

Customer Owned KMS Keys are 2 types

AWS Managed Customer owned KMS keys and Customer managed and owned KMS keys.

AWS Managed KMS keys are created when a service such as S3 use KMS for encryption

Customer managed KMS keys are created by customer.

Customer managed KMS keys are much more configurable, meaning we could allow cross account access

Can edit the key policy. Can allow other AWS accounts to access KMS keys to perform ops.

Both key types support key rotation.

Rotation is the process in which the physical backed key is changed.

With AWS managed KMS keys, the keys are rotated for every 365 days or 1 year and key rotation can’t be disabled.

With customer managed KMS keys, rotation is optional. If rotation is enabled then it can happen once an year.

The backing key, physical key material [ and all previous ones are stored, even after rotation so that data encrypted by older keys can be decrypted

A shortcut to a particular KMS key , called an alias, can be created, per region.

Neither aliases or keys are global

KMS has to be explicitly told that keys trust the AWS account they are in.

Key Policies and Security

Permissions on keys are handled in a separate manner

Many services will trust the account they are contained in. If we grant access via an identity policy, it is alllowed unless there’s an explicit deny

In KMS this trust is added explicitly added on a key policy or its not added

Starting point of kms keys are key policy - similar to resource policies.

Every KMS key has a key policy

Every customer managed key can have its policy changed

Unlike other services, KMS has to be told that the keys trust the account that they are part of.

We need key policy to grant access to a key using Identity services

Generally KMS is managed using key policies, trusting the account and then using identity policies to let IAM users interact with the key

In high security environments, this trust should be removed and permissions be added in the key policy
The IAM permissions for KMS are granular and split based on functions. This way the product admins or others aren’t given rights to access the data encrypted by KMS

Single PUT upload

This is the S3 Default upload method

The data is transferred in a single stream to S3

A file becomes an object and is uploaded using the PUT object API and happens in a single stream.

• The problem: If the stream fails, then the entire upload fails and the upload operation has to restart from beginning

  resulting in wastage of internet bandwidth and time.

• Whenever anything is downloaded, it is done on multiple streams

• Single stream of data is not reliable when data is transferred over long distances

• Speed and reliability are the limitations of a single stream of data.

When the data is transferred over two points the lowest of the speeds are selected.

Data transfer protocols like bit-torrent have been developed for speedy, distributed transfer of data.

If a single PUT upload is used, only 5GB data could be transferred.

The solution – multipart upload – improves speed and reliability, by data into individual parts

Multipart Upload

• Minimum size for multi-part upload is 100MB

• A multipart upload can be split into a max of 10000 parts and each part can be of size between 5MB to 5GB

• The last part is leftover and can be < 5MB

• Multipart upload is so effective because each part is treated as an individual upload.

If the part fails then only the failed part needs to be restarted. The risk significantly reduces.

The transfer rate of the entire upload is the sum of all individual parts.

S3 Transfer Acceleration

Distributed teams around the world can make use of the public internet to upload data to a bucket in any AWS region and we have no control over the path taken by the data as it can take an indirect path.

Transfer Acceleration uses network of AWS Edge locations

S3 bucket needs to be enabled for transfer acceleration.

By default, its switched off. There are some restrictions for enabling it.

Bucket name cannot contain periods and names should be DNS compatible

So the data is transferred to the nearest AWS edge location and from there the data is transferred over the AWS Global network, which tend to be direct links

The internet is a multipurpose public network built for flexibility and resilience not for speed.The AWS network is built to connect from region to region – much faster and lower latency (delay)

These objects can be referenced by their version ID to interact directly - or omit this to reference the latest version of an object

Objects aren't deleted - object deletion markers are put in place to hide objects.

Versioning is an essential feature to understand for the exam.

MFA Delete is a related feature which is also discussed.

Obj Versioning is controlled at bucket level.

Bucket at disabled state. But once the feature is enabled, it can’t be disabled again.

If needed the bucket can be suspended and can be re-enabled

Without versioning enabled, each obj. is identified using object key. ( that is, obj. name)

If you modify an obj. the original version of that document is replaced.

Versioning lets you store multiple versions of the object in a bucket.

When an object is modified by some operation, a new version is generated and it replaces the old one.

There is an attribute of the obj. known as ID. When versioning is disabled, the ID of the obj. is set to null. That’s the meaning of the versioning being off on a bucket – all objects will have id of null.

If versioning is enabled and new obj is added, then that obj. is given an ID e.g 11111

If any modifications are made to this object, the new version is given a new id and retains old version The newest or latest version is known as current version.

If an object is accessed without specifying the id then the current version is returned.

Versioning also affects deletions. If we indicate to S3 that we want to delete an object and not specify the version id then S3 adds a new special. version of the obj. known as delete marker.

The delete marker makes the obj look deleted but actually the obj is hidden.

So delete marker is a special version which hides all previous versions of the obj.

If delete marker is deleted then essentially undeletes the object making current version active again.

If an object is requested to be deleted by specifying the version id then the version is deleted.

If the current version is deleted, then the previous version becomes current ver.

Imp points to remember.

Once the versioning feature is enabled on a bucket, it cannot be disabled only suspended.

If versioning is enabled, then we have to pay the storage costs for all those versions.

If versioning is suspended, the above. point holds good

MFA delete

• It is enabled in the versioning configuration of bucket

• When enabled, MFA is required to change versioning state from enabled to suspended or vice-versa

• To delete versions also, MFA is required.

• How is it done – (Serial number of MFA with Code passed with the API calls)

For e.g the getobject API

This feature allows access via standard HTTP for blogs, static websites provided the identity trying to access it is authenticated and authorized, the get object API call is used to access those resources.

Using API calls is secure and flexible.

Using S3 for static website hosting makes it more useful. It allows access via browser using the HTTP/S protocol.

Doing it is simple: Set an index and error html document

If a specific page is mentioned then that page is accessed. If no specific page is mentioned, then index page is displayed.

When a static website is enabled, we have to point the index document at a specific object in the S3 bucket.

The error document is the same. Used when something goes wrong. Both files need to be html documents because static website delivers html pages

When the feature is enabled on a S3 bucket, a website hosting endpoint is created. It is a specific address that the bucket can be accessed from using HTTP

Exact name of endpoint is influenced by bucket name that is chosen and the region it is in.

Custom domain can be used via R53 if the name of the bucket name matches the domain.

Static web hosting is great for hosting blogs and also for two other scenarios:

i) Offloading

ii) Out of band pages

Offloading

If a website needs to display static html pages like images, those images could be moved into S3 bucket which has static website hosting enabled.

S3 is cheaper for storage and delivery of any media versus a compute service

Out of band pages

An example of this is maintenance page

For e.g if a maintenance page needs to be shown while the server is under maintenance then DNS could be changed and point customers at a backup static website hosted on S3 and it can provide this status message and/or maybe the details of biz support team

S3 Pricing

S3 has a simple pricing structure.

There is a per gigabyte per month charge for storing data in S3 and also for transfer out, there is a per gig month charge

There is no transfer fee for transferring data into S3

There is a charge for different ops on the per 1000 operation rate.

In the free tier plan, 5G of monthly storage is provided free of cost. 20k get and 20k put requests are free.

There is a charge for every 1000 operation on the Static website

The S3 service is private by default.The only identity that has any initial access to an S3 bucket is the account root user of the account which owns that bucket.

Any other identity has to be granted permission on that bucket explicitly

The first way of doing that is using an S3 bucket policy

An S3 bucket policy is a resource policy.

A resource policy is like an identity policy. But is attached to resources instead of identities.

– Provides a resource perspective on permissions

The difference b/w identity policy and resource policy

• Identity policy determines what the identity can access.

• Resource policy determines who can access that resource?

Identity policies can be attached to identities in one’s own account, can only control security in one’s own acct. No way of giving an identity in another account access to an S3 bucket

Different Uses of Bucket (Resource) Policies

Resource policies can allow identities from same or different account access over S3 buckets because the policy is attached to the resource and it can reference any other identities from same/different account

Another benefit: Can allow/deny anonymous principals(ones that are not auth. by AWS)

They can be used to block specific IP Addresses from accessing objects

In a bucket policy statement, there is a principal component. This is a way to differentiate an identity policy from a resource policy

Bucket policies can be used to allow or deny access from specific ip addresses

Bucket policy can be used to block access to a specific folder or prefix inside an S3 Bucket

ACLs

ACLs can be used to control access to subresources.

They are a legacy way of providing access. It is not recommended by AWS.
They have simple and inflexible permissions

It cannot be applied on a bunch of objects

Some important pointers – When to use identity policy versus bucket policy (resource policy)

Use identity policies if you want to provide access to multiple resources

If you want to control access from a single place like IAM, use identity policy

Use Resource policy - If you want to control access for a single product like S3

Use bucket(i.e resource) policy if you want to grant anonymous or cross-account access

AWS Control Tower offers a straightforward way to set up and govern an AWS multi-account environment, following prescriptive best practices. It is the evolution of AWS Organizations.

It orchestrates the capabilities of several other AWS services, including AWS Organizations, AWS Service Catalog, and AWS IAM Identity Center (successor to AWS Single Sign-On), to build a landing zone in less than an hour. Resources are set up and managed on your behalf. To help keep your organizations and accounts from drift, which is divergence from best practices, AWS Control Tower applies preventive and detective controls (guardrails). For example, you can use guardrails to help ensure that security logs and necessary cross-account access permissions are created, and not altered.

Parts of the Control Tower

Landing zone – A multi-account environment.

SSO/ID Federation, Centralized Logging and Auditing using a combination of services like CloudWatch, CloudTrail, AWS Config, SNS etc...

Guard rails (controls) – designed to Detect/mandate rules/standards across all accounts

Account Factory – A template which Automates and Standardises new account creation

Dashboard – single page overview of the entire organization

AWS Control Tower is created out of a standard account. This account becomes the management account of the landing zone. It contains Control Tower – which orchestrates everything

AWS Org – provides the multi-account structure like OUs and SCPs

SSO now known as IAM Identity Center - means we can use AWS internal identities or external federated IDs to access everything in the LZ that we have permissions to.

When it is setup the first time, it creates two OUs

1. A foundational OU – named security OU

2. A custom OU – named sandbox – used for less testing and less rigid security situations

In the foundational OUs two accounts namely Audit acct and Log Archive account are created

Log archive account

This is for users who need access to all logging information for all of the enrolled accounts in the landing zone.

E.g – AWS Config Logs, Cloudtrail logs

These logs are stored in this account so that they’re isolated. Need to explicitly grant access to this account Read only archive account for logging

Audit account

It is for users who needs access to audit information provided by Control Tower.

Any third party tool can be used for auditing the data stored in this account

Can also use SNS for notification about changes to any governance or security policies

And use CloudWatch for monitoring landing zone-wide metrics

There is also account factory – which can be likened to a team of robots which create, modify or delete AWS according to business needs.

We could interact with them from both Control Tower console or via Service Catalog

Account Factory

It will create as many accounts as needed in the Custom OU called sandbox in a fully automated way

Configuration of these accounts are handled by Account Factory

From account and networking perspective, the baseline or cookie cutter configs are applied thus ensuring a consistent config across all AWS accounts within landing zone

Control Tower uses CFN under the hoods for much of this automation

Control Tower uses both AWS Config and SCP to implement account guardrails which detect drifts or divergences from governance standards or prevent the drifts from happening in the first place.

The action of provisioning also includes automatic applying of guardrails

Account administration permission can be given to any named user – making it true self service

Account and network standard configurable applied according organizational requirements like IP addressing of VPCs

Accounts can be closed or repurposed.

This whole process can be fully integrated with business SDLC using API if we need accounts provisioned according to the stage of app development, client demos, or software testing.

Landing Zone

Landing zone – A feature designed for anyone to implement Well Architected multi-account environment.

Home Region – the region where the product is initially deployed into.

Landing Zone is built by services like AWS Config, AWS Organizations, CFN etc. So basically it is a product which orchestrates the features of various AWS services

Security OU – Log Archive and Audit Accounts (Cloud trail and Config Logs)

Custom (Sandbox) OU – Less rigid security. Used for testing purposes

Uses IAM Identity Center for single-sign-on, ID federation – using existing identity stores to access AWS accounts

Provides monitoring and notifications using CloudWatch and SNS

Allows end user account provisioning via Service Catalog

GuardRails

Basically rules for multi-account governance

Three different types – mandatory, strongly recommended or elective

Mandatory rules are also applied compulsorily enforced

Strongly recommended ones are strongly recommended rules are given by AWS

Elective are optional and are for niche requirements

Two functional types – Preventative and Detective

Preventative – stops us from doing things – implemented using Service Control Policies

Examples of enforced rules are allow/deny regions or disallow bucket policy changes within accounts inside the Landing Zone.

The second functional way is detective. They are like compliance checks.They use AWS Config rules to check whether configuration of a service is according to best practice.

Three states – clear, in violation or not enabled.

Preventative rules will stop things from happening.

Detective rules will identify things

So they are an important security and governance feature in AWS

Cloudtrail is a product which logs API calls/activities which affect AWS accounts as CloudTrail Events

It is a record of an activity in an AWS account.

It could be any action taken by a user, role or service like starting an EC2 instance, deleting an S3 bucket and so on.

It stores the events in the CloudTrail event history for the last 90 days by default w/o any costs

To customise CloudTrail for more than 90 days, a trail needs to be created.

Three types of events : Management events and Data events and Insight events

management events provide info about management operations that are performed on resources in your AWS account, a.k.a control plane operations e.g creating/terminating an EC2 inst. or creating VPC

Data events : Provide info about resource operations performed on or in a resource. e.g object uploaded to S3, accessing obj. in S3, invoking Lamda function

By default only management operations are logged or recorded because data events take up a high volume of storage very quickly. For example, logging every access to an S3 object

CT trail is a unit of configuration within the CT product. It is a way of providing config to CT on how to operate.

A trail logs event for the AWS region that it’s created in by default Global trail needs to be enabled by default

CT is a region-based service.

2 methods of config for operating CT: One region trail or All-region trail

A single region trail will be logged in the region it is created in.

- an all region trail functions as one logical trail. Any new region is added the cloud trail config is automatically updated.

Most services log events in the region where the event occurred.

A small number of services (IAM, STS or Cloudfront) log global events to one region and are known as global service events and to log events related to that, they need to enabled in that trail.

When a cloudtrail is created management events are captured and if enabled manually, data events are also captured.

We need to manually enable data event capturing. It is not enabled by default.

After creating the trail, the events are stored as compressed JSON log files in S3 in a compressed format.

These logs can be passed to any tool which is capable of reading and intepreting them.

CT can be integrated with CloudWatch logs. The data can be stored in it. The data then can be searched or assigned a metric and interpreted.

A recent feature added to the cloudtrail product is that an org trail can be created, if the trail is created from the management account of an org, then all information from all the member accounts in the org is stored in the CT product

Its a public service hosted in the AWS public zone. It’s accessible from AWS VPCs , on-premises environment or other cloud platforms provided you have network connectivity and AWS permissions.

The CW Logs product allows to store, monitor and access logging data

Logging data – is basically – piece of information and timestamp-DDMMYYhhmmss TZ format

It has integrations with AWS services like EC2, VPC Flow Logs, Lambda, CloudTrail, R53, etc..that means these services can store logging data in CW Logs' The security is provided using IAM roles or service roles

For anything outside AWS or for logging custom application or OS logs on EC2 – CW agent

There is a third way i.e the dev kits for AWS and implement logging into CW Logs directly from the application

CW Logs can take the logging data and generate a metric from it – known as a metric filter

Let’s suppose there’s a linux instance which has an OS log file which captures any failed ssh connection attempts

If that log from an instance is fed into CW Logs service, then a metric filter can scan the log data constantly and update the metric and an alarm can be setup based on the metric

CW Architecture

It is also a regional service

Starting point – Logging source which can include AWS products and services, external computer services , virtual or physical servers db, external APIs

These source feed data into CW Logs as log events

Log events have a time stamp and message block.

CW Logs treats it as a raw block of data but the data can be interpreted and columns and fields can be defined from that.

Log events are stored inside log streams and log streams – which are sequence of log events from same source

One log stream for one instance and one type of log

/var/log/messages for one instance is one log stream for system diagnostics in linux-based systems. Each log stream is an ordered set of log events for a specific source for a specific thing.

Log groups – container for multiple log streams for same type of logging data

Log groups – also stores configuration settings like retention settings and permissions

Retention settings and permissions are also defined on log groups – applicable to all log streams inside it

Metric filters will also be configured for log groups. Metric filters constantly review log events for any log streams in that log group looking for certain patterns, when found, the metric filters increment a metric and these metric can have alarms which would notify admins or AWS or external systems to take action. So it is a very useful products

There will be two OUs one for production and one for development and the member accounts will be put into their respective OUs.

SCP inherits down the organization tree if attached the org container then it affects entire organization. If attached to OUs it affects the member accounts in the OU Management account is special in that the SCP does not affect the management account I.e management account can’t be restricted using SCP attached to it directly or indirectly It is beneficial and limitation. Because of that management account should not be used to manage AWS resources

SCP are account permission boundaries

They limit what AWS acct can do.(including acct root user)

Account Root user can’t be controlled but using SCP what the acct can be done can be restricted and what the identities of account can do.

But if the account could be restricted then in effect, the account root user is also restricted. (indirectly)

For ex. An SCP could be used to restrict the usage of an acct outside a region, to restrict the size of the EC2 instance to be used within the acct.

They DON'T grant any permissions

They just act as a boundary of what is allowed and not allowed

Allow List vs Deny List

There are two ways of applying service control policies

Allow List - Block by default and allow the ones in the allow list

Deny List - Allow all services and block the ones in the deny list

Deny list is default

When SCPs are enabled on the AWS org. AWS apply a default policy which is called full AWS access, applied to all the organization and all OUs.

In the default implementation, SCP have no effect since nothing is restricted but there is an implicit deny like IAM policies, when an SCP is applied

So the same priority rules applied: Explicit Deny Explicit allow Implicit Deny

Any service explicitly allowed by scp can be granted access to identities within that account, unless there’s an explicit deny within an SCP then a service can’t be granted

Explicit deny always wins

And in the absence of either and also an explicit allow SCP policy, there will be an implicit deny

Allow list implementation is a two part architecture.

First part, remove AWS full access policy. So only the default implicit deny is present. Add any services that you want to allow into a policy

So irrespective of the identity permissions the identities in the account are provided, they are allowed only to access the services in the allow list. This is far more secure as we need to specify which services need to explicitly allowed.

But the admin overhead is more as services needed to be added to the allow list

How SCPs impact permissions in an AWS acct.

Only permissions which are allowed within identity policies in the account and are allowed by a service control policy are actually active.

Effective permissions for identities within an acct are the overlap between any identity policies and applicable SCPs

These are derived from manufacturing, high-reliability organizations and high trust management models

The Manufacturing Value Stream

One of the most basic concepts in Lean is that of the value stream. Per Karen Martin and Mike Osterling, it is the sequence of activities an organization undertakes to deliver upon a customer request. OR
the sequence of activities required to design, produce and deliver a good or service to a customer including the dual flow of information and material

In the manufacturing world, the value stream is clearly visible. The focus is on creating a fast and predictable lead time by creating a smooth work flow using techniques like

• small batch sizes

• reduce work in process

• prevent defects from being passed to downstream

The Technology Value Stream

In DevOps, technology value stream is the process required to convert a business hypothesis into a technology-enabled service or feature that delivers value to the customer.

The input is the business objective, idea which is accepted and added to the committed backlog of work. The dev teams follow agile or some other iterative process will likely convert it to user stories and feature specification.which is then implemented in code to the app or service. The code is then checked into a version control repository where the change is integrated and tested with the rest of the software system.
Since value is created when the app or service is running in production, we should ensure that the deployments are also performed without issues.

Focus on Deployment Lead Time

As described above, the value stream begins when any engineer(Dev, QA etc.) checks a change into the VCS and ends when the changes is running successfully in production, providing value to customer and is generating useful feedback and telemetry.

The first phase of work is similar to Lean Product Development whose process time could be varied.
The second phase which involves testing, deployment and operations is similar to Lean Manufacturing.

Instead of large batches of work processed sequentially, the goal is to have testing, deployment and ops happening at same time as with design/development. It is possible when work is done in small batches.

Lead Time vs Processing Time

There are two measures in Lean Manufacturing: Lead Time and Processing Time
Lead Time starts when request is made and ends when it is fulfilled
Processing Time begins only when the work is started and ignores waiting in the queue time
Achieving shorter lead time almost always requires reducing the wait-in-queue time.

The DevOps Ideal: Deployment Lead Times of Minutes

The deployment lead times in companies where DevOps practices are not adopted the lead times require months.

But in the devops environment, devs receive fast, constant feedback on their work, enabling them to quickly and independently implement, integrate and validate their code and deploy it in production.

Observing "%C/A" as a Measure of Rework

The third key metric or measure is percent complete and accurate which helps in measuring the quality of the output of each step in the value stream.

According to Karen Martin and Mike Osterling the percent complete and accurate can be obtained by asking downstream customers what percentage of the time they receive the work that is 'usable as is'

The Three Ways: The Principles Underpinning DevOps

The First Way enables fast left-to-right flow of work from Dev to Ops to the customer. In order to maximise flow:

• Make work visible

• Reduce batch sizes and work intervals

• Build in quality by preventing defects from being passed to downstream centers

• constantly optimize for global goals

The Second Way enables fast and constant flow of feedback from right to left all stages of our value stream. It needs faster detection and recovery helping to create work systems where problems are found and fixed before a disaster occurs.

The Third Way enables a generative high-trust culture that supports a dynamic, disciplined and scientific approach to experimentation and risk-taking, thus creating organizational learning from our successes and failures, helping the organization win in the marketplace.

“If you can’t explain something to a first year student, then you haven’t really understood.”

And also this:

“A great way to learn is to explain.”

So this blog as a whole, will attempt to test the understanding of yours truly with regards to the topics/concepts/ideas discussed and reinforce it and at the same time help you the reader in better comprehending them.

NB: Those of us who were following the earlier series regarding AWS SAA and The Devops handbook, Terraform etc., we hope to restart them as early as possible.

1. Public Cloud
2. Multi Cloud
3. Hybrid Cloud
4. Private Cloud

All major cloud service providers have a public and private version of the cloud computing platform. To know when and where to use them. We need to understand the difference between the different deployment models.

Public Cloud

This type of cloud environment/platform is available to the public. The examples are AWS, Azure and Google. This means these platforms follow 5 major features of the cloud computing and also that they are available to the public.

Multi Cloud

This cloud platform refers to storing company data in multiple public cloud. One part in AWS and the other in Azure.

Hybrid Cloud

The cloud infrastructure is a composition of two or more distinct cloud infrastructures (private, community, or public) that remain unique entities, but are bound together by standardized or proprietary technology that enables data and application portability (e.g., cloud bursting for load balancing between clouds).

Private Cloud

The cloud infrastructure is provisioned for exclusive use by a single organization comprising multiple consumers in an org. (e.g., business units). It may be owned, managed, and operated by the organization, a third party, or some combination of them, and it may exist on or off-premises

Note: Hybrid Cloud is NOT public cloud with legacy on-premises infrastructure

The book is divided into 6 parts containing a total of 23 chapters. In this post we will discuss a summary of the introduction to Part 1

Part 1 - The Three Ways

Let's see how different technology and management movements converged to set the stage for the DevOps movement.

In this part we will focus on the principles of Flow, Feedback and Continual Learning and Experimentation

But before that lets look at the factors that led to the formation of the DevOps movement, briefly.

The DevOps principles and practices are due to a convergence of many philosophical and management movements. These are the result of applying decades of lessons from the manufacturing, high reliability organizations, high trust management models and others like Lean, Theory of Constraints, Toyota Production System, resilience engineering, learning organizations, safety culture, human factors and many others like Agile to the IT value stream.

Lets discuss about of a few of them.

The Lean Movement

In the 1980s, techniques such as value stream mapping, kanban boards and total productive maintenance were codified for the Toyota Production System. In 1997, the Lean Enterprise Institute started researching applications of Lean to other value streams such as service industry and healthcare.

Two of Lean's central ideas are:

1. Manufacturing lead time - the time required to convert raw materials into finished goods - is the best predictor of quality, customer satisfaction and employee happiness.
2. The best predictor of short lead times is small batch sizes of work

The objectives of Lean principles are:

• Create value to customer through systems thinking
• Embracing scientific thinking
• Creating flow and pull
• Assuring quality at the source
• Leading with humility
• Respecting every individual

The Agile Manifesto

The Agile Manifesto was created in 2001 at an invite-only event attended by 17 experts in what were known as lightweight methods in software development with the motive to capture the advantages of these methods and codify them into a set of values

One key principle was to: "deliver working software frequently, from a couple of weeks to a couple of months with a preference to the shorter timescale" The other values focused on the need for small, self-motivated teams working in a high-trust management model. Agile has increased the productivity and responsiveness of many development organizations. Many key DevOps moments occured in the Agile Conferences like the following:

• At the 2008 Agile Conference in Toronto, Canada, there was a session held discussing the possibility of applying Agile principles to infrastructure (also referred to as "Agile system administration")
• At the 2009 Velocity Conference, John Allspaw and Paul Hammond delivered the now famous "10 Deploys per Day: Dev and Ops Cooperation at Flickr" presentation in which they discussed Creating shared goals for both Dev and Ops teams AND Using continous integration practices so that deployment is part of everyone's daily work

And then Patrick Debois organized the DevOpsDays conference in Ghent, Belgium in 2009 which witnessed the coining of the term "DevOps"

The Continous Delivery Movement

Jez Humble and David Farley built upon the development discipline of continous build, test and integration by extending the concept to continous delivery which defines the role about deployment pipeline to make sure code and infrastructure are always in a deployable state and that all code checked into the source control platform can be safely deployable.

Toyota Kata

In 2009 Mike Rother wrote Toyota Kata which talked about his twenty year journey to understand and codify the Toyota Production System. Now an interesting twist, he found that the companies who adopted the Lean approach failed to replicate the performance levels at the Toyota plants. He concluded that one important practice which he called the improvement kata, was missing. It needed daily habitual practice of improvement work

In the next post, we will discuss about value stream, applying lean principles to the technology value stream and The Three Ways of Flow, Feedback and Continous Learning and Experimentation.

Cloud Computing - what is it...really

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. This cloud model is composed of five essential characteristics, three service models, and four deployment models.

5 Characteristics that make any system cloud

On demand self service

A consumer can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with each service provider.

OR

The customer can provision capabilities (i.e features, products) as required automatically without needing human interaction

Broad network access

Capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, tablets, laptops, and workstations).

Resource pooling

The provider’s computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to consumer demand. There is a sense of location independence in that the customer generally has no control or knowledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter). Examples of resources include storage, processing, memory, and network bandwidth. Resources are pooled to serve multiple customers using a multi-tenant model This characteristic deals with abstraction and economies of scale

Economy of scale: The cost advantage that a company gains when it increases its production. As quantity of goods increase, the per unit cost the goods decreases. In cloud computing context, this means that the SP buys larger number of devices for a pool of customers and can maintain larger buffer for growth The customer can make use of the devices when needed and stop using it when there is lesser demand. The vendor or service provider manages the capacity But there is data isolation so that the data of one customer is not visible to another

Rapid elasticity

Capabilities can be elastically provisioned and released, in some cases automatically, to scale rapidly outward and inward commensurate with demand. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be appropriated in any quantity at any time.

OR

Capabilities(Resources) can be elastically provisioned and released to scale rapidly outward and inward with demand. To the customer the capabilities available for provisioning often appear to be unlimited. Cost also increases and decreases appropriately with demand. Elasticity also means under provisioning that leads to performance issues to customers and over-provisioning which leads to financial wastage, can be avoided.

Measured service

Cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency for both the provider and consumer of the utilized service.

OR

Resource usage can be monitored, controlled, reported and billed. Provide on-demand billing

This is for version 0.11 or earlier. An HCL file has blocks and arguments A block is defined within curly braces and contains arguments in key value pair format representing config data.

What information is kept in the block The platform and the resources in the platform First element is block called resource block id by word resource Then we have the resource type and then inside the resource block we have arguments in the form of key value pair

Simple Terraform workflow is as follows

1. Write the Terraform configuration file.
2. Run the init command
3. Review the execution plan using plan command
4. Once ready, apply changes using apply command

When we run the terraform init command, Terraform will check the configuration file and initialize the working directory containing the .tf file. It will understand the resource type and download the appropriate provider to work on the resources declared in the .tf file Now the resource can be created.. If we want to see the steps involved in the creation of the resource, run the terraform plan command. The + symbol signifies that resource will be created. Once we have verified all the steps, run the terraform apply command. It will show the execution plan once again and ask the user’s confirmation to proceed with the steps. Then run the terraform show command to view the details of the created resource. To know the resource types supported by a provider and the arguments needed, refer to the terraform documentation.

Update and Destroy Infrastructure

Once the necessary changes are done When the terraform plan is run, the output shows that resource will be replaced -/+ shows that the resource will be deleted and recreated Even though the change is small, Terraform will delete and recreate the file This infrastructure is called immutable infrastructure. To continue with the change, run the command terraform apply and type yes to confirm To destroy the infrastructure, run the command terraform destroy An execution plan is shown and when we give the confirmation yes, terraform proceeds to perform the action.

Major advantage - it be used to deploy infrastructure on multiple platforms like Cloud service providers, Vmware and other vendors

How does it manage infrastructure on third party platforms - through providers (API) It supports hundreds of providers and can work with almost any platforms

The language it uses – Hashicorp Configuration Language – a simple declarative language used to define infrastructure resources to be provisioned as blocks of code. The config file has a extension of .tf

[Note: HCL is used in version 0.11 and earlier]

It is an easy to read, write and understand even for a beginner

What is declarative mean? The code we define is the state that we want the infrastructure to be in.

How does Terraform move from current state to desired state? There are three phases of operation

• Init --→ It initializes the project and identifies the providers for the target environment.
• Plan --→ Drafts a plan to move from current state to desired state.

• Apply --→ Performs the necessary changes on target environment to the desired state

  If the target environment shifts from the desired state for any reason, the subsequent apply moves the tgt environment to desired state by fixing the missing part.

Any object that Terraform manages is called a resource. Manages the lifecycle of the resource from provisioning, managing to decommissioning.

It records the state of the infrastructure as it is in the real world and based on it it determines what action is needed while updating resources on a certain platform. It can ensure that the infrastructure is always in the desired state.

State ---→ blueprint of infrastructure deployed by Terraform.

It can read attributes of existing infrastructure by configuring data sources and can be used by Terraform to configure other resources.

Terraform Cloud and Terraform Enterprise provide additional features for simplified collaboration between teams managing infrastructure, improved security and centralized UI for managing TF deployments.

These features make it an excellent infrastructure provisioning tool.

The Traditional Way of managing Infrastructure.

App delivery in a traditional IT environment

The management of the business organization wants to roll out an application

They give the requirements for the application

The business analyst gathers needs and comes out with a set of high-level technical requirements.

These are passed to the Solution Architect who comes up with the architecture for the deployment of the application like the type, specs and count of servers needed for front end web serves, backend web servers, databases, loadbalancers

Traditionally this would be deployed in the on-prem environment (using resources in the datacenter)

For any additional hardware, the procurement team places an order with the vendor who’d purchase and deliver to the datacenter from anywhere between a few days, weeks or months.

Once the hardware is received, the field engineers from the client are responsible for the setting up the hardware on a rack. ( rack and stack)

The sys admins do the initial configs on the systems and network engineers make them available on the network.

Storage admins assign storage to the servers

The backup admins configure backups.

Once the systems are setup as per the standards, they are then handed over to the app teams for deploying the apps.

This deployment models has quite a few disadvantages.

1. High turn around time. (From weeks to months)
2. Slower scaling time (Both scale up and scale down)
3. Overall cost to maintain this model quite high
4. Limited automation - Some aspects can be automated while others like rack and stack, cabling and deployment are manual and slow
5. Chances of human error high as many teams are engaged in many tasks which result in inconsistent environment.
6. Another major disadvantage – under-utilization of compute resources.

So organizations have been moving to virtualization and cloud computing to take advantage of cloud services

Overall time to setup infrastructure and Time to Market of Apps are significantly reduced

As datacenter and h/w are managed by the cloud service providers Infrastructure costs to manage h/w resources are reduced.

Cloud platforms support APIs, thus opening a huge opportunity for automation.

Built-in scaling capabilities helps to reduce resource wastage.

Cons of Web based Cloud platforms

We could provision infrastructure using the web interface provided by the platform however, it is not efficient in case of larger number of resources

Still the infrastructure needed to be handed over to various teams for application setup giving way for human errors.

Companies started to solve this using scripts and other tools for infrastructure provisioning and managing resources and environments. These became the IAC

Types of IAC Tools

One way of infrastructure provision is using the web console of cloud platforms or the provisioning process can be codified using scripts and programming languages

Thus we can write and execute code to define, provision, configure, update and eventually destroy infrastructure resources. This process or concept is known as Infrastructure as Code (IaC)

Using this concept almost any infrastructure component can be managed as code.

IAC has evolved to defining infrastructure using simple, human readable and high level language

IAC tools are designed with a specific goal in mind. Based on these, there are three types of IAC tools

• Configuration management
• Server templating
• Provisioning tools

Features of Configuration management tools are:

They are designed to install and manage software into existing infrastructure resources viz. Servers, db, networking devices etc.

Unlike shell scripts, these tools maintain a consistent and std structure of code making it easier to manage and update them as needed.

Can run or execute on multiple remote resources at once.

Can be checked or saved into a VCS allowing to distribute and reuse

Most important feature - They are idempotent ie they can run multiple times and each time it will only make changes necessary to bring the environment to a desired state. - meaning it’ll leave anything already in place as it is.

Features of Server templating tools

Examples are docker, packer and hashicorp vagrant

What do they do – create custom images of a virtual machine or container containing all necessary software and dependencies installed on them

Eliminates the need for installing software on the deployed virtual machine or container. Examples images on docker hub, custom images in AWS ECR etc.

They promote immutable infrastructure – means once a vm or container is deployed, it usually remains unchanged.

If changes are to be made to images then changes are made to the images and a new instance is deployed using the updated image. rather making changes to running instances as in the case of config. mgmt tools.

Features of Provisioning tools

They are used to provision infrastructure components like servers, db, subnets etc using simple declarative code.Examples are Terraform, AWS CloudFormation etc.

Terraform is vendor or platform agnostic and support provider plugins for all major cloud computing providers.

• using it in scripts
• while adding new services into the IT environment
• during change activities which require stopping and disabling of services

This is available in all Linux systems having systemd as their system and service manager.

To start and enable a systemd service

systemctl enable service --now

To stop and disable a systemd service

systemctl disable service --now