Correctly granting access to your Microsoft Azure resources is a process filled with potential pitfalls. Magnus Mårtensson, Azure MVP from Devoteam M Cloud, tells you how to do it the right way.

A big challenge for any company that uses Microsoft Azure cloud services is to grant access to your cloud resources in Azure in an appropriate and safe manner. It adds to the challenge that the administrative owner of the Azure accounts may be a less technical person who is responding to requests from ‘the tech people’ asking to be “unblocked to do their job”. 

The immediate risk is that the admin then grants full access to Azure “to be done with it” and intends to let the tech department sort itself out! Already, too many people have the wrong type of access. The potential issues that can arise from such a situation includes, but are not limited to, wasteful Azure spend, mistakes that impact production (and ultimately your customers), and security or data breaches.

I am here to tell you that using Azure to grant access to Azure may not be as straightforward as you would think. Why not? Read on and I will lay out the challenge and how it might be mitigated for your company.

The dual nature of identity and access for Azure

If the question is “who has access to what in Azure?” the answer is split in two: In Azure, access to your resources is granted to users originating from your Azure Active Directory (AAD). Azure and AAD are two different services. Do you see a potential challenge here? 

The problem, of course, becomes the division between the two systems. For instance, you cannot ask AAD what a user has access to in Azure when that information is stored in Azure. Likewise, you cannot ask Azure who is behind an identity because that information is stored in AAD. As you probably know, the names of people who have access in Azure are shown in the Azure portal. Did you also know that these names have been fetched by Azure from the AAD?

Access to one thing does not mean access to all

I will, for context, give a few examples on specific access grants stored in Azure. Access could, for instance, be permission to read settings on an Azure Web App resource. Access could be to manage (or contribute to) a storage account resource. A side is that data about a resource and data inside the same resource are two different things in Azure. Access can often be granted to manage a data storage resource but not at the same time granted to the data inside it. One could also have access to the data plane only (the data content) of the same resource, but not to manage the resource itself. It is possible to have both types of access at the same time, but those can be different grants.

The last initial thing worth noting is that Azure Active Directory is the world’s leading provider of identity and access management. Naturally, we want to take advantage of this strength when managing access to your critical Azure resources!

Beware of granting ghosts and zombies access in your Azure!

I want to tell you a story to illustrate some of the key points of access management. Meet “Casper” and “Dave”. They work at your company where they are part of your Azure team.

Casper and Dave need access to Azure resources to do their job. For this reason, both Casper and Dave have been granted access to a resource group.

As happens sometimes, both Casper and Dave later leave the company. The AAD account for Casper’s identity is erased. Casper is now in theory entirely gone – and consequently he ghosts the AAD. Dave‘s account, however, is NOT erased but instead just suspended, so Dave can no longer sign in. 

In both cases Casper and Dave no longer have access to your Azure resources, which is good. What is not good is the state of access in Azure now that they have left. Complications arise immediately when a new manager comes onboard.

An access nightmare

Meet “Sarah”. Sarah is a new manager in your company, and she joins the Azure team. Sarah needs to understand who has access to various Azure resources. She looks in Azure at the resource group where Casper and Dave had access and she is immediately confused.

It seems like there is a ‘ghost’ haunting the Azure resource group! Some unknown entity seems to have lingering access. Even if Azure asks the AAD which identity it is, the response will be that it is unknown since Casper’s identity does not exist in the AAD anymore. 

Afraid of what this might mean, Sarah reaches out to a person who has access in the same group, Dave. Worry builds as Dave does not answer Sarah’s messages. As we know, Dave left but when you look in access management for the resource group, it looks like he is still roaming around in Azure (almost like a zombie). Like with Casper, Azure asks the AAD who has this specific access. And the response will be Dave – even though Dave’s account is locked, and Dave no longer works in the company.

Sarah is now very confused and does not know what to do. Can we help Sarah?

Let’s do better!

This weird little story is too close to reality for many companies I am called in to work with. The current approach does not work well and we instead need to try something different. I want to be very clear that the method I am discussing next is the method we have employed to manage access to IT systems with Active Directory for decades. All I propose is that we take the same care in Azure, which unfortunately we as an industry, currently do not.

Why is has it become so bad?

A legacy reason for poor access control in Azure is that before AAD was available for access control in Azure, the only option was to access Azure subscriptions and resources using a Microsoft Account from Microsoft. Access to a subscription was completely binary. Either you did not have access to all the resources in the subscription, or you had full access to all of them. Remnants of this legacy still exist today but should be avoided and removed if you ever see it. The right way is to use AAD users and corporate accounts.

A more likely reason for the degraded state affairs for access control in Azure is instead related to a general inexperience in Cloud. Many of us are learning Azure while working in it, and access control is the first pitfall we stumble into. After a while, when we begin to understand what access might and should be in Azure, we find it hard to start over. On occasion I have been asked to reset access for users of Azure. That often leads to interesting conversations about why it is necessary to remove access previously granted. An employee might ask: “Am I not trusted anymore?”. 

The real stance must be that we wish to unburden our co-workers with the responsibility of too much access. Restricting appropriate access where it has been too lax is not a punishment, but a gift. Most people I speak to do not want too much access, only the access they need to do their job, and only while they are performing a work task that requires access.

Granting and managing access to Azure at scale: How to do it correctly

We need to use the full potential of the AAD, which is its tools for managing users and groups of users. Then we use a naming convention and automation tools to ensure the right group in AAD has the right access in Azure. We want to only grant access in Azure to Active Directory Security Groups and never grant access directly to AAD identity objects, such as machine accounts or human users!

Returning to the story, Sarah has employed two new engineers, “Cecilia” and “Daisy”, and proceeds to grant them access to some company Azure resources. Sarah creates an AAD Security Group which she grants access in Azure. She makes Cecilia and Daisy members of that group. Now, the engineers have membership granted in AAD in a manageable way, and only groups have access to Azure. AAD knows which users’ identities are members in what security contexts. When that needs to change, this is managed in the AAD.

Technical or administrative skills?

As hinted at just above, while it is important to always grant appropriate access, just-in-time when needed, and with least possible privilege, it is imperative to also be empowered to do that with uniformity and in the correct way each time. We discussed how it is common that an admin setting up an Azure commitment for the company is not necessarily experienced in setting up appropriate access. 

But why is the technical administrator for an Azure setup not doing it, then? The administrators are often not accustomed to Azure yet, and even when they are, the right access at the right time is not always something they have/or take the time for. 

They may become an unwilling bottleneck in the process, slowing down progress, and blocking employees from accessing Azure when they rightfully need to. Is it better then to grant full access to some team members? No, it is worse!

Invest in access!

The right access to Azure is a fundament worth investing in, so that you can feel secure as a company and grant your employees the freedom to be accountable for only the things they need to work with, while not needing to be bothered with any undue access they never asked for!

Keeping a tight ship while cruising toward the cloud, while challenging and uncomfortable at first, will be greatly rewarded down the line!

Originally posted at: https://www.devoteam.com/expert-view/vanquishing-azure-access-ghosts-and-zombies/

I want to share with you my approach to setting appropriate access in Azure and delegate specific access control to team leaders. (Original post on LinkedIn)

When granting access to Azure for users I observe that appropriately tight control over access is brushed over and mismanaged. Often too much access, such as owner, is granted, and to way too many people. One longer-term consequence is that the team eventually spend time searching for weird bugs that “only happen in production”, just to find – much later and under pressure from dissatisfied users and screaming managers – that the real issue was that production and test was not configured the same way. Of course, automated deployments go a long way to mitigate environment drift, but the challenge with too much access, or rather inappropriate access, is that people with too much access can go willy-nilly and modify things they thought was right. This works well until it is not. Errors will always happen when humans are involved. Another issue with too much access is compliance. It is hardly compliant to allow the whole team into production environments with permanent all access.

Privileged Identity Management

This article is not about Privileged Identity Management (PIM), but I figured I would mention that as well since PIM is great technology. With PIM a user may be eligible to have a certain access, but it does not hold that privilege on a permanent basis. PIM may be configured to require managerial approval for activation. This is great because then someone with appropriate responsibility allows critical high privilege access and it is always time limited. If an account is compromised that has PIM configured for all high privilege access that account has much less value for the intruder. In relation to PIM, I advise my customers to enforce an audit after each access elevation to investigate the purpose of the required elevation and to document what was modified in the accessed environment.

The way I design appropriate and professional access to the Azure environments that belong to a team or a service is to limit scope and grant more access in development compared to production. Typically, any service hosted in Azure has multiple environments correlating to the stage it is in. The notion of “DTAP” – Development, Test, Acceptance and Production is very common. I grant the whole team read access on all environments – so that they can see all the resources they are responsible for. I only grant contributor access to development and test. For acceptance and production environments I only accept access via automated deployments using a Service Principal. Deployments must be automated for production. Acceptance is the final staging ground before production, so it is treated the same way. To acceptance and production, it is appropriate to configure PIM as discussed above to potentially allow humans to modify these environments only after appropriate access elevation.

Avoid subscription level access

I do not create a new Azure Subscription for each new service environment, because I think that is just a very big and chunky thing to deliver. It is more than what was asked for and therefore inappropriate to give. If a team receives their own subscription with full access, they tend to become careless with resources and wasteful practices ensue. The only thing you get in access control if you create more subscriptions is – more subscriptions. There is no inherent value at all for access and security. A definite downside to granting access to a whole subscription is that the access granted is not only to all resource groups and resources in the subscription at this time. Access is also granted to future resource groups that will be created. I find that the scope of resource group for access is more appropriate and relates more closely to the required access for a scenario. Subscription scope leads to the mistake of granting too much access in the future.

Azure Role Based Access Control

Role Based Access Control (RBAC) in Azure allows to assign access using a certain role to a certain scope. For example, the whole team may be readers in the service environment, but only Alice and Bob may be contributors. A scope can be a resource, a resource group or a subscription. In fact, a scope may also span multiple subscriptions. There are multiple challenges with assigning direct access to users in Azure. I will discuss them below while instead applying access via security groups.

Access using Azure AD Security Groups

When I grant access in Azure via RBAC I avoid granting it directly to humans or to service principals. Instead, I create Azure AD (AAD) security groups – called Access Groups (AG). To them I grant the access of a specific role in a specific environment. For example, foo_p_contributor_AG might be a security group in AAD for contributors on the production environment for a service called Foo. I find that putting something unique, the service name, as the prefix works well with searching in AAD. Search for that prefix and you find all AAD objects that relate to it. I proceed to grant this security group the access role of contributor in Azure for the resource group for the service foo. At this point you might think – well this is nothing new at all Magnus, it is the way access has been granted in IT forever. I agree, it is. It is just that for some reason I have yet to find an Azure project in the wild that adheres to this reasonable standard. Instead, what I see is that whole teams are owners of whole subscriptions. And mayhem ensues.

There is another excellent reason for using AG’s to grant access in Azure. Those AAD objects are security groups that are specific to the resource groups in Azure. They exist while their target resource groups exist and are only used for that context. If you grant RBAC to an AAD object in Azure and later that object is removed, the access granted remains in Azure. Azure can no longer look up in the AAD what the access was granted to. This creates an access ghost in Azure. Here you see I have granted reader access to the user Demo-Dave (he works in my office) and to a demo security group:

If I delete “temporary demo group” from my AAD the picture looks like this:

Access is granted in Azure to an object which is unknown in the AAD. I call these access ghosts.

Furthermore, if Demo-Dave leaves the company and his account is suspended access looks like this:

Yes, the same. I have visited many companies over the years where I have noted that “I probably need to talk to Dave because he has access to this service” only to find out that “Oh, Dave doesn’t work here anymore, he quit six months ago”. Azure still remembers Dave.

Azure AD Security Group Owners

Another thing that is quite nice when it comes to AAD groups is that they can have owners.

Group owners may add and remove members in the group and even add and remove other group owners. This feature creates the opportunity to effectively delegate to team leaders the authority to add and remove people to contribute to the services they manage. In Azure RBAC the same requires the owner role for the scope. The owner role allows the user to grant any role to anyone, not only selected roles. This is again inappropriate and ill advised. By setting up a security group owner for the Access Group you delegate only the ability to grant the same access to Azure that has been granted to the group. This is usually exactly appropriate for the context.

The path to improve your security posture and control in Azure

I urge everyone to invest in Azure access control and do the following:

• Add appropriately scoped access groups with roles to your service environments in Azure! Do this when you create the service environment – Reader and Contributor is a great start and it covers about 95% of your use cases. Other roles can be granted upon request and involves setting up another access group.
• Avoid granting access to subscription scope or higher! Instead, grant access to resource group scopes. Granting access to resource scope only is often too fine-grained. I prefer to segment applications over multiple resource groups with related scopes such as foo_web and foo_db.
• Remove direct human and service principal RBAC assignments from Azure! Instead, make the same members of the appropriate access groups in AAD.
• Treat your service environments in Azure as the resource group AND the collaborating AAD access groups! Remove the access groups if the Azure resource is removed.

Following this approach, you are far less likely to ever hunt environment drift bugs, there will not be any ghosts in your RBAC assignments and there will also not be any access granted to people who no longer works in the company. When you get into delegating access control via security group owners you will discover how great it is and what a wonderful yet appropriately scoped empowerment it is.