Server Man
Well-Known Member
- May 17, 2015
- Windows 10
- Chrome 67.0.3396.99
During the May 2018 Community Call and a tweet a few weeks later, we mentioned that PowerShell team was spending significant time in the Windows codebase. We even demoed using the Active Directory PowerShell Module from PowerShell Core 6 during the PowerShell Community Call.
We started investigating some of the top requested modules that ship with Windows to see how to get them working in PowerShell Core 6. Our eventual goal is to have near 100% parity with Windows PowerShell for the in-box modules, but we expect this effort to span across multiple releases of Windows as there are many modules and PowerShell team is working with each feature team on an individual basis to port and validate their modules. The good news is that you can pick up newly compatible modules with frequent updates to Windows 10, and Windows Insiders will see them even sooner.
TL;DR; (Too Long; Didn’t Read)
Starting with Windows 10 Insiders build 17711 or newer, you can use many Windows PowerShell modules with PowerShell Core 6.1-Preview.4 and if you see an issue or difference compared to Windows PowerShell 5.1, please open an issue in the PowerShellModuleCoverage repository.
The Journey to Porting Windows PowerShell modules
Prioritizing Windows PowerShell Modules
Windows ships with many PowerShell modules. We created a prioritized list of Windows PowerShell modules to investigate compatibility and portability based on customer and partner requests.
We ended up with a list of what we thought would have maximum customer impact. For example, the top requested module was the Active Directory PowerShell Module, so we made sure to partner with the AD team and investigate their module first. After investigation, we also deferred work on certain modules that were infeasible to port to PowerShell Core given our time frame and expertise.
ApiPort
First, we looked at was using ApiPort to identify APIs used by Windows PowerShell modules that were not available in .NET Core. Depending on the API, we were able to include additional .NET Core compatible assemblies, and in others, we rewrote the code to use a different API that is functionally equivalent. In some cases, the API was simply unavailable, so we could only mark the module as incompatible by setting CompatiblePSEditions to just Desktop in the module manifest. The initial investigation showed that >95% of the APIs used by Windows PowerShell modules were available in .NET Standard (or compatible assemblies).
.NET Standard 2.0
.NET Standard is a formal specification of .NET APIs intended to be available on all .NET implementations. For example, .NET Framework 4.6.1+ used by Windows PowerShell and .NET Core 2.0+ used by PowerShell Core 6 both implement APIs specified by .NET Standard 2.0. By rebuilding existing source code for Windows PowerShell modules to target .NET Standard instead of .NET Framework, we would get compile-time errors if an API wasn’t available and shouldn’t be used.
The Windows build environment is self-contained and did not support building code targeting .NET Standard. The next work item was to work with the folks on the Windows Engineering System team to enable building managed assemblies and target .NET Standard 2.0. They were excellent partners, and together, we built a simple PowerShell module against .NET Standard 2.0 that worked in both Windows PowerShell 5.1 and PowerShell Core 6.
Windows Compatibility Pack
Although the proof of concept to use the Windows build environment to produce a .NET Standard assembly worked, actual Windows PowerShell modules used many APIs beyond just what is in the formal .NET Standard 2.0 specification (and associated reference assembly). To help developers move from .NET Framework to .NET Core, the .NET team published a set of assemblies called the Windows Compatibility Pack as an extension to .NET Standard. The Windows Compatibility Pack brought back important namespaces like System.DirectoryServices, that Windows PowerShell modules required.
To make this work with PowerShell Core 6, we added the Windows Compatibility Pack 2.0.0 assemblies into PowerShell Core 6.1. This also means that any module author who wants to leverage the Windows Compatibility Pack will not need to redistribute the Windows Compatibility Pack assemblies with their modules, as they are now built into PowerShell Core 6.1 (.NET Framework would already have its own implementations of the same APIs).
Trouble in Paradise
Things were moving along getting the infrastructure ready for the porting work. However, we were also up against a strict timeline. We made a tactical decision to minimize risk of not getting our changes into Windows by moving away from targeting .NET Standard 2.0 and Windows Compatibility Pack explicitly. Building against .NET Standard 2.0 and relying on Windows Compatibility Pack meant we had to ship the Windows Compatibility Pack facade assemblies in Windows so that .NET Framework knew how to find the actual implementation assembly to work correctly. Adding new assemblies in the Windows build environment is easy compared to adding new assemblies into Windows itself. Adding new binaries to Windows require making changes to specific files in the code base and you can only validate it worked successfully when you have a complete build of Windows. Windows code base is huge and takes a long time to build on a developer machine. A complete build may not be successful due to other changes happening in the overall code base which is what happened to us.
In addition, we also hit issues where other assemblies in the Windows code base had a dependency on the Windows PowerShell module assembly (like a graphical user interface sitting on top of PowerShell) and would break the build since the GUI code built with .NET Framework didn’t know how to reference a .NET Standard built assembly.
Since we could not successfully get a working build of Windows with our assembly additions, we decided to abandon that work and go without .NET Standard compilation.
Moving Forward
Because .NET Standard is an API contract (like a C/C++ header file), we did not need to formally rebuild Windows PowerShell modules with .NET Standard to have them work in .NET Core and thus PowerShell Core 6. What .NET Standard gives us is some assurance that future additions to Windows PowerShell modules would catch new API usage that is not part of .NET Standard or Windows Compatibility Pack. However, although .NET Standard indicates that an API is available, we found out during testing that even if the API exists, it may throw PlatformNotSupportedException at runtime which prevents the Windows PowerShell module from working correctly in PowerShell Core 6. As we found these, we’ve had to make code changes in the modules to handle these cases appropriately during runtime whether it’s running within Windows PowerShell or PowerShell Core 6.
Changes in PowerShell Core 6.1
In anticipation of the changes we were making in Windows PowerShell modules, we also needed changes in PowerShell Core 6.1 to make the experience seamless. In mid-June we published an RFC to get community feedback on the user experience we would enable for using compatible Windows PowerShell modules from PowerShell Core 6.
Basically, with PowerShell Core 6.1-Preview.4 the PowerShell engine will look in $env:WinDirSystem32WindowsPowerShellv1.0Modules as part of enumerating available modules from $env
SModulePath. However, by default, only Windows PowerShell module manifests that explicitly indicate Core as a supported in the CompatiblePSEditions property. Windows PowerShell module manifests that don’t have the CompatiblePSEditions property or only has Desktop will not show up as an available module by default. The -SkipEditionCheck switch on Import-Module can be used to override this behavior. It is important to reiterate that this logic only applies to modules in $env:WinDirSystem32WindowsPowerShellv1.0Modules path. If you used Install-Module to install a module from the PowerShell Gallery and its module manifest either doesn’t contain CompatiblePSEditions nor includes Core, the PSEdition check logic doesn’t apply and that module will show up as available since it was explicitly installed from PowerShell Core 6.
Availability of changes in Windows
The Windows PowerShell module changes we’ve made will start showing up in Windows 10 Insiders Build 17711. This means that with that build of Windows 10 and PowerShell Core 6.1-Preview.4 you can start using some of the Windows PowerShell modules we’ve identified as Core compatible. A reminder that as we continue to make changes in Windows, they will show up in newer builds of Windows 10 Insiders. Our target is to get >65% of the Windows PowerShell in-box modules compatible with PowerShell Core 6.1 within the next Windows 10 release. We’ll continue to work with Windows partner teams to get the number of compatible modules closer to 100%.
Community Feedback
We work with individual Windows teams who are the owners of their modules to validate our changes as much as possible with PowerShell Core 6. However, due to how they wrote their tests, it’s not as simple as rerunning their existing tests within PowerShell Core 6 and in many cases, we opted to do some limited manual validation. This means that although we’ve made some effort to validate Windows PowerShell modules marked as compatible with Core, we cannot guarantee they are fully compatible (see the note about runtime issues above) and we would treat them as bugs.
Here the community can help start using Windows PowerShell modules in PowerShell Core 6.1-Preview.4 and if you see an issue or difference compared to Windows PowerShell 5.1, please open an issue in the PowerShellModuleCoverage repository. We are only using that repository for issue tracking. If someone else has opened an issue on a module (or identified a module they need but isn’t ported yet), please give it a ?? (thumbs up) to up-vote the issue which helps us prioritize the work.
Note that if there are other Windows PowerShell modules by Microsoft that aren’t in-box in Windows that you think is a priority to port, please open an issue in that repo! PowerShell Core compatibility will help us with engagements with those partner teams if we can show them the demand for it.
Windows Compatibility Module
As we continue this journey porting Windows PowerShell modules which I expect to take multiple Windows releases, you can use any Windows PowerShell module from PowerShell Core 6 without waiting for a port. A significant majority of the Windows codebase is written in C/C++ (95+%!). A number of teams who are primarily native code developers wrote their PowerShell modules in Managed C++. However, there are currently no plans to support Managed C++ in .NET Core. For these sets of modules and those waiting to be ported, we published the Windows Compatibility module which leverages implicit remoting from PowerShell Core 6 to Windows PowerShell, so it makes the cmdlets from that module available in PowerShell Core 6 but executes it within Windows PowerShell.
You can use this today with the stable release of PowerShell Core 6.0.3 and doesn’t require using PowerShell Core 6.1-Preview builds. If you find issues with this module, please open issues in the Windows Compatibility repo.
Summary
The initial goal of the PowerShell Core 6.0 release was to provide a cross-platform automation platform. I expect by end of July, we should hit a new high of 3 million startups of PowerShell Core 6!
Approximately 80% of the usage is on Linux, so I consider us working with the community to have achieved our first goal.
The goal of PowerShell Core 6.1 release (targeting end of August for General Availability) is to help move Windows PowerShell users forward to PowerShell Core 6 by providing compatibility with in-box Windows PowerShell modules that they depend upon. It is important to understand that the Windows PowerShell module porting work won’t be complete by the time PowerShell Core 6.1 GA nor when the next version of Windows 10 is released, and we expect to continue this work as needed to eventually get complete coverage.
Thanks to all the feedback and contributions from the community helping PowerShell move forward!
Steve Lee
Principal Software Engineering Manager
PowerShell Team
Azure Compute
Steve Lee (@Steve_MSFT) | Twitter
Continue reading...
We started investigating some of the top requested modules that ship with Windows to see how to get them working in PowerShell Core 6. Our eventual goal is to have near 100% parity with Windows PowerShell for the in-box modules, but we expect this effort to span across multiple releases of Windows as there are many modules and PowerShell team is working with each feature team on an individual basis to port and validate their modules. The good news is that you can pick up newly compatible modules with frequent updates to Windows 10, and Windows Insiders will see them even sooner.
TL;DR; (Too Long; Didn’t Read)
Starting with Windows 10 Insiders build 17711 or newer, you can use many Windows PowerShell modules with PowerShell Core 6.1-Preview.4 and if you see an issue or difference compared to Windows PowerShell 5.1, please open an issue in the PowerShellModuleCoverage repository.
The Journey to Porting Windows PowerShell modules
Prioritizing Windows PowerShell Modules
Windows ships with many PowerShell modules. We created a prioritized list of Windows PowerShell modules to investigate compatibility and portability based on customer and partner requests.
We ended up with a list of what we thought would have maximum customer impact. For example, the top requested module was the Active Directory PowerShell Module, so we made sure to partner with the AD team and investigate their module first. After investigation, we also deferred work on certain modules that were infeasible to port to PowerShell Core given our time frame and expertise.
ApiPort
First, we looked at was using ApiPort to identify APIs used by Windows PowerShell modules that were not available in .NET Core. Depending on the API, we were able to include additional .NET Core compatible assemblies, and in others, we rewrote the code to use a different API that is functionally equivalent. In some cases, the API was simply unavailable, so we could only mark the module as incompatible by setting CompatiblePSEditions to just Desktop in the module manifest. The initial investigation showed that >95% of the APIs used by Windows PowerShell modules were available in .NET Standard (or compatible assemblies).
.NET Standard 2.0
.NET Standard is a formal specification of .NET APIs intended to be available on all .NET implementations. For example, .NET Framework 4.6.1+ used by Windows PowerShell and .NET Core 2.0+ used by PowerShell Core 6 both implement APIs specified by .NET Standard 2.0. By rebuilding existing source code for Windows PowerShell modules to target .NET Standard instead of .NET Framework, we would get compile-time errors if an API wasn’t available and shouldn’t be used.
The Windows build environment is self-contained and did not support building code targeting .NET Standard. The next work item was to work with the folks on the Windows Engineering System team to enable building managed assemblies and target .NET Standard 2.0. They were excellent partners, and together, we built a simple PowerShell module against .NET Standard 2.0 that worked in both Windows PowerShell 5.1 and PowerShell Core 6.
Windows Compatibility Pack
Although the proof of concept to use the Windows build environment to produce a .NET Standard assembly worked, actual Windows PowerShell modules used many APIs beyond just what is in the formal .NET Standard 2.0 specification (and associated reference assembly). To help developers move from .NET Framework to .NET Core, the .NET team published a set of assemblies called the Windows Compatibility Pack as an extension to .NET Standard. The Windows Compatibility Pack brought back important namespaces like System.DirectoryServices, that Windows PowerShell modules required.
To make this work with PowerShell Core 6, we added the Windows Compatibility Pack 2.0.0 assemblies into PowerShell Core 6.1. This also means that any module author who wants to leverage the Windows Compatibility Pack will not need to redistribute the Windows Compatibility Pack assemblies with their modules, as they are now built into PowerShell Core 6.1 (.NET Framework would already have its own implementations of the same APIs).
Trouble in Paradise
Things were moving along getting the infrastructure ready for the porting work. However, we were also up against a strict timeline. We made a tactical decision to minimize risk of not getting our changes into Windows by moving away from targeting .NET Standard 2.0 and Windows Compatibility Pack explicitly. Building against .NET Standard 2.0 and relying on Windows Compatibility Pack meant we had to ship the Windows Compatibility Pack facade assemblies in Windows so that .NET Framework knew how to find the actual implementation assembly to work correctly. Adding new assemblies in the Windows build environment is easy compared to adding new assemblies into Windows itself. Adding new binaries to Windows require making changes to specific files in the code base and you can only validate it worked successfully when you have a complete build of Windows. Windows code base is huge and takes a long time to build on a developer machine. A complete build may not be successful due to other changes happening in the overall code base which is what happened to us.
In addition, we also hit issues where other assemblies in the Windows code base had a dependency on the Windows PowerShell module assembly (like a graphical user interface sitting on top of PowerShell) and would break the build since the GUI code built with .NET Framework didn’t know how to reference a .NET Standard built assembly.
Since we could not successfully get a working build of Windows with our assembly additions, we decided to abandon that work and go without .NET Standard compilation.
Moving Forward
Because .NET Standard is an API contract (like a C/C++ header file), we did not need to formally rebuild Windows PowerShell modules with .NET Standard to have them work in .NET Core and thus PowerShell Core 6. What .NET Standard gives us is some assurance that future additions to Windows PowerShell modules would catch new API usage that is not part of .NET Standard or Windows Compatibility Pack. However, although .NET Standard indicates that an API is available, we found out during testing that even if the API exists, it may throw PlatformNotSupportedException at runtime which prevents the Windows PowerShell module from working correctly in PowerShell Core 6. As we found these, we’ve had to make code changes in the modules to handle these cases appropriately during runtime whether it’s running within Windows PowerShell or PowerShell Core 6.
Changes in PowerShell Core 6.1
In anticipation of the changes we were making in Windows PowerShell modules, we also needed changes in PowerShell Core 6.1 to make the experience seamless. In mid-June we published an RFC to get community feedback on the user experience we would enable for using compatible Windows PowerShell modules from PowerShell Core 6.
Basically, with PowerShell Core 6.1-Preview.4 the PowerShell engine will look in $env:WinDirSystem32WindowsPowerShellv1.0Modules as part of enumerating available modules from $env
SModulePath. However, by default, only Windows PowerShell module manifests that explicitly indicate Core as a supported in the CompatiblePSEditions property. Windows PowerShell module manifests that don’t have the CompatiblePSEditions property or only has Desktop will not show up as an available module by default. The -SkipEditionCheck switch on Import-Module can be used to override this behavior. It is important to reiterate that this logic only applies to modules in $env:WinDirSystem32WindowsPowerShellv1.0Modules path. If you used Install-Module to install a module from the PowerShell Gallery and its module manifest either doesn’t contain CompatiblePSEditions nor includes Core, the PSEdition check logic doesn’t apply and that module will show up as available since it was explicitly installed from PowerShell Core 6.Availability of changes in Windows
The Windows PowerShell module changes we’ve made will start showing up in Windows 10 Insiders Build 17711. This means that with that build of Windows 10 and PowerShell Core 6.1-Preview.4 you can start using some of the Windows PowerShell modules we’ve identified as Core compatible. A reminder that as we continue to make changes in Windows, they will show up in newer builds of Windows 10 Insiders. Our target is to get >65% of the Windows PowerShell in-box modules compatible with PowerShell Core 6.1 within the next Windows 10 release. We’ll continue to work with Windows partner teams to get the number of compatible modules closer to 100%.
Community Feedback
We work with individual Windows teams who are the owners of their modules to validate our changes as much as possible with PowerShell Core 6. However, due to how they wrote their tests, it’s not as simple as rerunning their existing tests within PowerShell Core 6 and in many cases, we opted to do some limited manual validation. This means that although we’ve made some effort to validate Windows PowerShell modules marked as compatible with Core, we cannot guarantee they are fully compatible (see the note about runtime issues above) and we would treat them as bugs.
Here the community can help start using Windows PowerShell modules in PowerShell Core 6.1-Preview.4 and if you see an issue or difference compared to Windows PowerShell 5.1, please open an issue in the PowerShellModuleCoverage repository. We are only using that repository for issue tracking. If someone else has opened an issue on a module (or identified a module they need but isn’t ported yet), please give it a ?? (thumbs up) to up-vote the issue which helps us prioritize the work.
Note that if there are other Windows PowerShell modules by Microsoft that aren’t in-box in Windows that you think is a priority to port, please open an issue in that repo! PowerShell Core compatibility will help us with engagements with those partner teams if we can show them the demand for it.
Windows Compatibility Module
As we continue this journey porting Windows PowerShell modules which I expect to take multiple Windows releases, you can use any Windows PowerShell module from PowerShell Core 6 without waiting for a port. A significant majority of the Windows codebase is written in C/C++ (95+%!). A number of teams who are primarily native code developers wrote their PowerShell modules in Managed C++. However, there are currently no plans to support Managed C++ in .NET Core. For these sets of modules and those waiting to be ported, we published the Windows Compatibility module which leverages implicit remoting from PowerShell Core 6 to Windows PowerShell, so it makes the cmdlets from that module available in PowerShell Core 6 but executes it within Windows PowerShell.
You can use this today with the stable release of PowerShell Core 6.0.3 and doesn’t require using PowerShell Core 6.1-Preview builds. If you find issues with this module, please open issues in the Windows Compatibility repo.
Summary
The initial goal of the PowerShell Core 6.0 release was to provide a cross-platform automation platform. I expect by end of July, we should hit a new high of 3 million startups of PowerShell Core 6!
Approximately 80% of the usage is on Linux, so I consider us working with the community to have achieved our first goal.
The goal of PowerShell Core 6.1 release (targeting end of August for General Availability) is to help move Windows PowerShell users forward to PowerShell Core 6 by providing compatibility with in-box Windows PowerShell modules that they depend upon. It is important to understand that the Windows PowerShell module porting work won’t be complete by the time PowerShell Core 6.1 GA nor when the next version of Windows 10 is released, and we expect to continue this work as needed to eventually get complete coverage.
Thanks to all the feedback and contributions from the community helping PowerShell move forward!
Steve Lee
Principal Software Engineering Manager
PowerShell Team
Azure Compute
Steve Lee (@Steve_MSFT) | Twitter
Continue reading...