Directive: Use Gears and Goggles

The Cybersecurity and Infrastructure Security Agency (CISA) published Binding Operational Directive (BOD) 25-01 on December 17, 2024, which ordered federal agencies to secure Microsoft 365 tenants over the period of February through June of 2025. 

CISA had already open-sourced the SCuBAGear tool, which can check 130 security checks covering Microsoft Entra ID and five applications: Defender, Exchange, PowerPlatform, Sharepoint, and Teams. Included are checks such as:

• MS.AAD.5.2v1: Only administrators SHALL be allowed to consent to [OAuth] applications
• MS.SHAREPOINT.2.1v1: File and folder default sharing scope SHALL be set to Specific people (only the people the user specifies).
• MS.TEAMS.1.3v1: Anonymous users and dial-in callers SHOULD NOT be admitted automatically.

CISA also released in late 2024, SCuBAGoggles, for checking Google Workspace organizations and applications.

The Good

This is a much needed benchmark from CISA, a reputable authority. Although BOD 25-01 applies to federal agencies, there is a good chance that commercial organizations will adopt the baseline policies as has been seen in the past with publications by other federal agencies such as NIST and 800-53.

We have already had numerous best practices, security policies, and benchmarks, including detailed work from the Center for Internet Security (CIS) with their Microsoft 365 and Azure Foundation Benchmarks. What sets CISA BOD 25-01 apart is that it is more than just a set of recommendations. With an open source reference implementation provided, it empowers organizations to get started ASAP i.e., there is no need to wait for a vendor to deliver these security checks.

The SCuBAGear code is written in PowerShell with a fairly practical design and implementation where data collection of the Microsoft application objects and settings is separated from the security check evaluation logic, which is written in Rego rules with an Open Policy Agent (OPA) implementation. 

• Each test is well thought out and written. 
• Like other security check frameworks, there is explanation of “why” the control is necessary or needed as well as remediation guidance. The CIS benchmarks have traditionally been the standard for prescriptive guidance and SCuBAGear is in the same category.
• The included sample reports are what any Entra or Google admin would want to get started on hardening their tenants or applications. 

The Gaps

First, a number of the 130 checks cannot be checked programmatically through an API nor can they be checked via a browser, so organizations how many checks they are performing.

Second, and more importantly, some analysis is usually required to understand the coverage with respect to other benchmarks like CIS or to determine whether there is overlap or gaps if multiple checks are being run.

Next, while prevention is critical, detection is even more important.

“Prevention is critical but detection is mandatory” – Eric Cole

Security configuration checks of any kind are good practices but like every defensive layer, are imperfect. Implementation of SCuBA will help prevent threats and abuse, but without complementary detection controls, organizations will not know if their security configurations are sufficient or have been bypassed, leaving them exposed to higher risk.

SCuBA focuses on security configuration (posture) checks and does not account for actual usage (real threat, abuse, exploit). One needs to supplement configuration checks with active threat detection, which is crucial for distinguishing theoretical threats vs actual threats in action.

It is one thing if the side window of the proverbial house is left open, but a whole another level of threat if a stranger opens that window and climbs through. Similarly, within SCuBAGear, there is a risk if Legacy Authentication (MS.AAD.1.1v1) is allowed, but the risk is higher if activity is detected that exploits those older protocols, especially by users accessing applications that contain sensitive data.

Even if static configuration checks pass, there is risk that attackers bypass the prevention measures that were checked. In other words, even if all the windows and doors are locked, interior cameras are required at a minimum, to accurately assess risk for active threats.

"Murphy's Law: If you set it, it will mutate. Immediately."

Mistakes happen, prevention is Imperfect, and configurations drift. In the most optimistic well-run security organizations, the intended governance or security policy implemented as a baseline set of controls will deviate from the actual controls and measures in production. There has to be a tight feedback between secure baselines and governed usage and detection of baseline drift and vulnerabilities. 

Every prevention or posture setting is ideally covered by a detection that identifies any deviation or drift from the intended setting. This will catch accidental misconfigurations by security administrators or perhaps more malicious actions by bad actors attempting to loosen controls as part of an attack.

"History whispers, but we often wear noise-canceling headphones."

Posture raises similar issues as Vulnerability Management twenty years ago. With VM, it’s easy to generate todo lists of fixes (e.g. 5000 hosts with 10 patches each). It is not easy to know if you are reducing risk (is it the same hosts, is risk concentrated in certain OSes, locations or admin groups, are we seeing recidivism). 

With cloud and SaaS posture, and with the scalable dynamic nature of cloud environments, it’s easy to generate a large report of misconfigured controls. But it’s much more difficult to address those and reduce risk. Are we seeing risks or violations concentrated in certain apps, security domains, under certain administrative groups, and over time, are the failed checks in the same area or exhibiting the same root cause?

“That’s in the ‘done’ folder”

Everyone has some level of detection capabilities, sometimes with a SIEM and worst case, the logs. Isn’t this just tilting at windmills? The key point is not to just have a detection solution, but to ensure that your detection solution is integrated and complementing your security checks.

This problem is far from being “done.” Most organizations struggle to:

• Know if the prevention measures are effective (detect config drift) – does reality match intention?
• Know if real threats are exploiting potential misconfiguration? The open window vs stranger already entered through that window.
• Understand why an abuse/attack occurred
• Perform RCA (risk cause analysis)
• Identify the most effective way to reduce that risk in the future (e.g, fix misconfigurations, focus on risky users, adopt mitigating measures) in order to close the risk loop.

Going Forward

All of these topics are important to not just understanding your risk from a security posture view, but whether it is practically affecting you, and importantly, measuring and understanding the cause of the risk in order to more permanently prevent it going forward.

There are concrete steps that will help most organizations:

• Implement your posture, whether it’s with CIS, SCUBA, or other. 
• SCUBA is excellent because it has a good reference implementation, and having that overcomes the barrier of how.
• Don’t stop at prevention/posture
• Implement a detection solution that
  
  • Detects configuration drift from the baseline
  • Refines risk based on real activity that exploited misconfiguration
  • Closes the loop by measuring and analyzing the risk over time to ensure that security domains, valued assets, privileged and non-privileged users, managed and unmanaged identities are addressed if they are the cause or concentration of that risk