Server Man
Well-Known Member
- May 17, 2015
- OS X
- Firefox 59.0
After running the PAW TAP program on the solution explained in this blogpost, I received tons of interests and great feedback. While the team is investigating on a plan, a lot of customers are asking how they can deploy PAW in their datacenter. This blogpost is dedicated on this topic.
To put the solution into perspective, PAW is just one piece to the privileged account protection puzzle, important but there are a lot more to consider. Microsoft has published a whitepaper which provides a much more comprehensive view. Even on the PAW topic, the whitepaper covers more aspects than this blogpost, such as creating different management tiers for your assets to reduce risk. This blogpost only focusses on one aspect, which is the PAW deployment, including the backend servers. I highly recommend you get familiar with the strategy explained in the whitepaper first before planning for PAW deployment.
Solution overview
There a few different options to deploy PAW, in this blogpost, I’ll focus on the solution which was evaluated in the PAW TAP program. The general feedback was positive, and customer liked the singled device configuration.
The solution leverages the shielded VM built in Windows 10 1709 to run secure workload, it includes the client configuration (end user device) and server backend. Here is a simplified topology overview:
A common misconception about PAW is “the device which the admin connects to, to get to the backend server (PAW?)”; in fact, PAW is the device which admin uses with physical keyboard and mouse. Consider the following diagram, PAW is commonly considered to be the device in the middle:
In this example, PAW refers to the User device , because you are using the keyboard/mouse on this device. If this device is compromised, attacker can get easily get the information to breach the datacenter.
Infrastructure deployment
When we designed the solution, one criteria is to fit it into the existing deployment. If you are familiar with the EASE, this is the model we tested to with the solution.
At a high-level, you will deploy a Host Guardian Server (HGS) in the environment. HGS is designed to run in its own forest, you can find details of the HGS deployment here.
In addition to HGS, PAW devices need to be managed. There are two common choices: leverage Azure (AAD and Intune) ; or using domain management. Furthermore, with on-prem deployment, a common question is about which domain should the PAW device joining to? Let’s start with identify the computer entities: physical PAW device, a desktop VM and one or more PAW VMs. The example here again using ESAE model as the example:
A few other infrastructure services are necessary for PAW but not dedicated for PAW:
Some services Azure offers alternative to, such as ATP which can do better security monitoring using a large database tracking many attack signatures.
Another important aspect to consider is the “Break glass” scenario. This is not particular to the PAW deployment. As with any process, if you require all the admins to use PAW, if the PAW device breaks, how would the admin work? Can they physically access the backend servers? Or are there backup devices? This should be considered as part of PAW deployment plan.
PAW device configuration
From end user perspective, the solution looks like:
A single device with multiple VMs running on it, with the following software/hardware requirements:
For the PAW device, I separated the partitions for the host OS and the VM VHDs. This way, if the host needs to be rebuilt, the VM information stays. In the past, you need to remember suspend BitLocker of the data drive if it’s host OS is being re-installed, and re-enable it afterwards; the latest release has better Bitlocker support where you don’t need to suspend. But of course, you should ensure there is a Bitlocker recovery plan in place. See here for more options on Bitlocker management.
The storage is partitioned as the following:
In my testing, I assigned 60GB to the host OS volume, and use the rest of the disk for the data volume. You can adjust the size depending on if you need to store certain data on the host OS. Given the workloads are moved to the VMs, ideally there is little workload and data to be stored on the host OS, so the volume size should be configured for just running the OS. In general, you will need to reserve some space for Windows update on the host OS.
Network
In the Windows 10 1709 release, there is a new type of VM switch “default switch” when Hyper-V is enabled. It allows virtual machines to share the host’s network connection. Without getting too deep into networking, this switch enables the VMs access to the host’s network whether you’re connected to WIFI, or Ethernet. It’s not manageable through the Hyper-V manager app. You can download the cmdlet module here to create additional VM INS switches.
To lockdown the network access through Edge, 1709 release also introduced Application Guard, you can take advantage of it to control the access. There is another option to do network-whitelisting through a proxy server, that’s how we do it in Microsoft. I think each has its own advantages. Proxy server network whitelisting is more mature, and there are several software vendors offering the solution, so I’m not going to go into the details here. I’ll focus on the configuration leveraging the Windows Defender Application Guard (WDAG).
You can use group policy settings to configure WDAG, in my testing, I’ve configured the following settings on the host:
The configuration for these settings are straightforward, you can simply enable the policies. This will configure the policy on the default switch, to allow/block traffic appropriately.
The highlighted policy is the setting you can whitelist Urls. You can find more reference on how to configure the policy here. On my PAW device, I configured the following value for this policy: pawhgs.xxx.microsoft.com (this is the Url for the HGS server which the PAW devices attest to). If user tries to connect to any other Urls, it will be launched through WDAG.
Another common question is on VPN. When the host is required to connect through VPN, how can the desktop VM be configured to not use the host VPN? You can configure the host with a registry key to prevent the VM using the host VPN.
Note: Reboot the machine after the change.
Also, this VPN behavior will be improved in the future release.
VM management
In general, there should be little VM management which the end user needs to do. Majority of the work is for the IT admins who provisions the PAW devices, which includes creating the VMs. Currently, the VM management is supported through PowerShell. Inbox Hyper-V module can do almost all the VM management, shielded VM provisioning scenario is supported by the GuardedFabricTools our team released in the Gallery.
Host user start menu customization
You can use group policy to change the start menu layout on the PAW device. Here is an example I created to show only Edge in the start menu:
<LayoutModificationTemplate xmlns:defaultlayout="http://schemas.microsoft.com/Start/2014/FullDefaultLayout" xmlns:start="http://schemas.microsoft.com/Start/2014/StartLayout" Version="1" xmlns="http://schemas.microsoft.com/Start/2014/LayoutModification">
<LayoutOptions StartTileGroupCellWidth="6" />
<DefaultLayoutOverride>
<StartLayoutCollection>
<defaultlayout:StartLayout GroupCellWidth="6">
<start:Group Name="PAW">
<start:Tile AppUserModelID="Microsoft.MicrosoftEdge_8wekyb3d8bbwe!MicrosoftEdge"
Size="2x2"
Column="0"
Row="0"/>
<start
esktopApplicationTile DesktopApplicationLinkPath="DawFilesbinStartProgramVMManager.lnk"
Size="2x2"
Column="2"
Row="0"/>
<startesktopApplicationTile DesktopApplicationID="{1AC14E77-02E7-4E5D-B744-2EB1AE5198B7}WindowsPowerShellv1.0powershell.exe"
Size="2x2"
Column="4"
Row="0"/>
</start:Group>
</defaultlayout:StartLayout>
</StartLayoutCollection>
</DefaultLayoutOverride>
</LayoutModificationTemplate>
You can save the content above to an xml file (startmenu.xml), and use group policy to apply it:
Computer Configuration > Administrative Templates >Start Menu and Taskbar
Enable the Start Layout policy and specify the startmenu.xml file full path. Check the docs for more details.
After VMs provisioned, you can also add additional RDP connection to the start menu, so user can easily launch them. For details on how to create more short cuts in the start menu, see here.
Image rollout
Windows 10 has a number of security features which can better protect the operating system. I wrote a separate blog post on how to enable them. The question I often get is “how should I deploy the device?” Again, there are a couple of options, both approach should follow the clean source principle:
Note that, to comply with the clean source principle, you should ensure using the separate deployment infrastructure from the existing one used for production. This is to ensure a higher trust level for PAW.
I think it’s also possible to do all the configuration through Intune, but I’m not the expert on Intune, I’m still looking for help to get all the CSPs created to customize the host. It will be a more scalable approach for future.
VM provisioning
You will need shielded VM templates to provision the PAW VMs. Template creation is a separate topic covered here, and I also shared an example of creating PAW VMs with Kiosk experience.
Once you have created the templates, you can store it centrally, and download it to the PAW device for PAW VM provisioning.
Desktop VM images are regular VHDs, you can simply download it locally to the device and create the VM from it. A common question is whether the desktop VM should be shielded; in my view, it should be managed in a similar secure level as other desktops in the environment. Host device has Bitlocker enabled, which also protects the desktop VM as a regular desktop. Shielding desktop VM will provide another layer of protection to prevent physical access to the device, and copying the desktop VM to another machine. If you do want to shield the desktop VM, one key scenario is to allow user to start the desktop VM while the device is offline. This is a new feature added in the 0418 release.
Logon and Connect
In the previous Infrastructure deployment section, we looked at how each machine entity is managed by the backend, let’s take a look how user will connect to them here:
Physical host Desktop VM PAW VM
Logon user identity ESAE domain user Production domain user Privileged account for backend management
Logon method
(Note: Hello for business is not supported for VMs today)
Connect
Old fashion user name/password still works for all the connections, but they are less secure comparing to smartcard or Windows Hello for business.
Conclusion
Usually with deployment guide, I’d provide detailed step-by-step instructions. I’m trying a different approach to cover more on planning considerations and referencing to other docs for the detailed instructions. It is such a large topic, I’m sure there are aspects which didn’t get covered, as always, feel free to post your questions/comments below or to us
Continue reading...
To put the solution into perspective, PAW is just one piece to the privileged account protection puzzle, important but there are a lot more to consider. Microsoft has published a whitepaper which provides a much more comprehensive view. Even on the PAW topic, the whitepaper covers more aspects than this blogpost, such as creating different management tiers for your assets to reduce risk. This blogpost only focusses on one aspect, which is the PAW deployment, including the backend servers. I highly recommend you get familiar with the strategy explained in the whitepaper first before planning for PAW deployment.
Solution overview
There a few different options to deploy PAW, in this blogpost, I’ll focus on the solution which was evaluated in the PAW TAP program. The general feedback was positive, and customer liked the singled device configuration.
The solution leverages the shielded VM built in Windows 10 1709 to run secure workload, it includes the client configuration (end user device) and server backend. Here is a simplified topology overview:
A common misconception about PAW is “the device which the admin connects to, to get to the backend server (PAW?)”; in fact, PAW is the device which admin uses with physical keyboard and mouse. Consider the following diagram, PAW is commonly considered to be the device in the middle:
In this example, PAW refers to the User device , because you are using the keyboard/mouse on this device. If this device is compromised, attacker can get easily get the information to breach the datacenter.
Infrastructure deployment
When we designed the solution, one criteria is to fit it into the existing deployment. If you are familiar with the EASE, this is the model we tested to with the solution.
At a high-level, you will deploy a Host Guardian Server (HGS) in the environment. HGS is designed to run in its own forest, you can find details of the HGS deployment here.
In addition to HGS, PAW devices need to be managed. There are two common choices: leverage Azure (AAD and Intune) ; or using domain management. Furthermore, with on-prem deployment, a common question is about which domain should the PAW device joining to? Let’s start with identify the computer entities: physical PAW device, a desktop VM and one or more PAW VMs. The example here again using ESAE model as the example:
- PAW physical device should be joined to the bastion forest. The risk for placing PAW in the production domain is the exposure to attack. I’ve seen breach to the SCCM server in production environment which leads to compromise of all the PAW devices when PAWs are managed by it. To reduce this risk, consider creating a separate PAW domain or if your company is open to cloud, PAW physical devices can also be AAD managed. We might create ESAE/PAW offering leverage AAD in the future;
- PAW VM is dedicated to manage a certain datacenter assets, it should be placed in the same secure bastion forest as the host;
- The desktop VM should be managed like all other user desktop machines, which joins to the production domain.
A few other infrastructure services are necessary for PAW but not dedicated for PAW:
- WDS: service to image the PAW devices
- SCCM: manage/configure PAW devices
- Monitoring service
- VPN server
- WSUS for patch management
Some services Azure offers alternative to, such as ATP which can do better security monitoring using a large database tracking many attack signatures.
Another important aspect to consider is the “Break glass” scenario. This is not particular to the PAW deployment. As with any process, if you require all the admins to use PAW, if the PAW device breaks, how would the admin work? Can they physically access the backend servers? Or are there backup devices? This should be considered as part of PAW deployment plan.
PAW device configuration
From end user perspective, the solution looks like:
A single device with multiple VMs running on it, with the following software/hardware requirements:
- License: To create shielded VMs, you need to enable “Guarded host” feature in Windows optional components. It is available on Windows client Enterprise edition and an E5 license.
- The device must have TPM2.0, running UEFI with secure boot enabled.
- Recommending memory 16GB or above; hard disk is 500GB and above to support multiple VMs.
For the PAW device, I separated the partitions for the host OS and the VM VHDs. This way, if the host needs to be rebuilt, the VM information stays. In the past, you need to remember suspend BitLocker of the data drive if it’s host OS is being re-installed, and re-enable it afterwards; the latest release has better Bitlocker support where you don’t need to suspend. But of course, you should ensure there is a Bitlocker recovery plan in place. See here for more options on Bitlocker management.
The storage is partitioned as the following:
In my testing, I assigned 60GB to the host OS volume, and use the rest of the disk for the data volume. You can adjust the size depending on if you need to store certain data on the host OS. Given the workloads are moved to the VMs, ideally there is little workload and data to be stored on the host OS, so the volume size should be configured for just running the OS. In general, you will need to reserve some space for Windows update on the host OS.
Network
In the Windows 10 1709 release, there is a new type of VM switch “default switch” when Hyper-V is enabled. It allows virtual machines to share the host’s network connection. Without getting too deep into networking, this switch enables the VMs access to the host’s network whether you’re connected to WIFI, or Ethernet. It’s not manageable through the Hyper-V manager app. You can download the cmdlet module here to create additional VM INS switches.
To lockdown the network access through Edge, 1709 release also introduced Application Guard, you can take advantage of it to control the access. There is another option to do network-whitelisting through a proxy server, that’s how we do it in Microsoft. I think each has its own advantages. Proxy server network whitelisting is more mature, and there are several software vendors offering the solution, so I’m not going to go into the details here. I’ll focus on the configuration leveraging the Windows Defender Application Guard (WDAG).
You can use group policy settings to configure WDAG, in my testing, I’ve configured the following settings on the host:
- Enable WDAG (Computer configurationAdministrative TemplatesWindows ComponentsWindows Defender Application guard)
The configuration for these settings are straightforward, you can simply enable the policies. This will configure the policy on the default switch, to allow/block traffic appropriately.
- Configure network isolation policy (Computer configurationAdministrative TemplatesNetworkNetwork Isolation)
The highlighted policy is the setting you can whitelist Urls. You can find more reference on how to configure the policy here. On my PAW device, I configured the following value for this policy: pawhgs.xxx.microsoft.com (this is the Url for the HGS server which the PAW devices attest to). If user tries to connect to any other Urls, it will be launched through WDAG.
Another common question is on VPN. When the host is required to connect through VPN, how can the desktop VM be configured to not use the host VPN? You can configure the host with a registry key to prevent the VM using the host VPN.
- Path: HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServiceshnsParameters
- Name: ICSFlags
- Type: DWORD
- Value: 1
Note: Reboot the machine after the change.
Also, this VPN behavior will be improved in the future release.
VM management
In general, there should be little VM management which the end user needs to do. Majority of the work is for the IT admins who provisions the PAW devices, which includes creating the VMs. Currently, the VM management is supported through PowerShell. Inbox Hyper-V module can do almost all the VM management, shielded VM provisioning scenario is supported by the GuardedFabricTools our team released in the Gallery.
Host user start menu customization
You can use group policy to change the start menu layout on the PAW device. Here is an example I created to show only Edge in the start menu:
<LayoutModificationTemplate xmlns:defaultlayout="http://schemas.microsoft.com/Start/2014/FullDefaultLayout" xmlns:start="http://schemas.microsoft.com/Start/2014/StartLayout" Version="1" xmlns="http://schemas.microsoft.com/Start/2014/LayoutModification">
<LayoutOptions StartTileGroupCellWidth="6" />
<DefaultLayoutOverride>
<StartLayoutCollection>
<defaultlayout:StartLayout GroupCellWidth="6">
<start:Group Name="PAW">
<start:Tile AppUserModelID="Microsoft.MicrosoftEdge_8wekyb3d8bbwe!MicrosoftEdge"
Size="2x2"
Column="0"
Row="0"/>
<start
esktopApplicationTile DesktopApplicationLinkPath="DawFilesbinStartProgramVMManager.lnk"Size="2x2"
Column="2"
Row="0"/>
<start
esktopApplicationTile DesktopApplicationID="{1AC14E77-02E7-4E5D-B744-2EB1AE5198B7}WindowsPowerShellv1.0powershell.exe"Size="2x2"
Column="4"
Row="0"/>
</start:Group>
</defaultlayout:StartLayout>
</StartLayoutCollection>
</DefaultLayoutOverride>
</LayoutModificationTemplate>
You can save the content above to an xml file (startmenu.xml), and use group policy to apply it:
Computer Configuration > Administrative Templates >Start Menu and Taskbar
Enable the Start Layout policy and specify the startmenu.xml file full path. Check the docs for more details.
After VMs provisioned, you can also add additional RDP connection to the start menu, so user can easily launch them. For details on how to create more short cuts in the start menu, see here.
Image rollout
Windows 10 has a number of security features which can better protect the operating system. I wrote a separate blog post on how to enable them. The question I often get is “how should I deploy the device?” Again, there are a couple of options, both approach should follow the clean source principle:
- Windows Deployment Server (WDS): you can first build the host OS iso image using MDT (Microsoft deployment toolkit), then using the WDS to enable device to install the image;
- USB boot: if the deployment is small, in my test environment, I created USB boot key with the ISO image created by MDT, and manually installed the PAW devices.
Note that, to comply with the clean source principle, you should ensure using the separate deployment infrastructure from the existing one used for production. This is to ensure a higher trust level for PAW.
I think it’s also possible to do all the configuration through Intune, but I’m not the expert on Intune, I’m still looking for help to get all the CSPs created to customize the host. It will be a more scalable approach for future.
VM provisioning
You will need shielded VM templates to provision the PAW VMs. Template creation is a separate topic covered here, and I also shared an example of creating PAW VMs with Kiosk experience.
Once you have created the templates, you can store it centrally, and download it to the PAW device for PAW VM provisioning.
Desktop VM images are regular VHDs, you can simply download it locally to the device and create the VM from it. A common question is whether the desktop VM should be shielded; in my view, it should be managed in a similar secure level as other desktops in the environment. Host device has Bitlocker enabled, which also protects the desktop VM as a regular desktop. Shielding desktop VM will provide another layer of protection to prevent physical access to the device, and copying the desktop VM to another machine. If you do want to shield the desktop VM, one key scenario is to allow user to start the desktop VM while the device is offline. This is a new feature added in the 0418 release.
Logon and Connect
In the previous Infrastructure deployment section, we looked at how each machine entity is managed by the backend, let’s take a look how user will connect to them here:
Physical host Desktop VM PAW VM
Logon user identity ESAE domain user Production domain user Privileged account for backend management
Logon method
- Windows Hello for business
- Smartcard
(Note: Hello for business is not supported for VMs today)
Connect
- Interactive logon
- RDP is disabled
Old fashion user name/password still works for all the connections, but they are less secure comparing to smartcard or Windows Hello for business.
Conclusion
Usually with deployment guide, I’d provide detailed step-by-step instructions. I’m trying a different approach to cover more on planning considerations and referencing to other docs for the detailed instructions. It is such a large topic, I’m sure there are aspects which didn’t get covered, as always, feel free to post your questions/comments below or to us
Continue reading...