Troubleshooting Hybrid Azure AD Join issue

I've been working with a customer this week to configure Hybrid Azure AD Join and co-management. In the first phase (HAADJ), most of my test devices successfully registered in Azure AD. However there were some stragglers. These devices didn't even show up as "Pending" in Azure AD. They just didn't appear at all.

It was time for troubleshooting. What is the best way to find out what is wrong? There are many things you can check.

Service Connection Point

As some of the domain joined devices had successfully registered in Azure AD already, it seemed unlikely that there was a problem with the Service Connection Point. However I had a look at it.

Beginning with version 1.1.819.0, Azure AD Connect provides you with a wizard to configure hybrid Azure AD join. Azure AD Connect deploys a Service Connection Point (SCP) into your Active Directory environment. A service connection point in AD is essentially an object that points to a specific service. The Azure AD Service Connection Point includes information on the following items in its keywords attribute:

• azureADId; The Azure Active Directory tenant ID
• azureADName; The Azure Active Directory tenant’s verified custom DNS domain name, or the *.onmicrosoft.com DNS domain name if no verified custom DNS domain name exists for the Azure AD tenant

 The Service Connection Point format is as follows:

CN=62a0ff2e-97b9-4513-943f-0d221bd30080,CN=Device Registration Configuration, CN=Services, CN=Configuration,DC=domain,DC=tld

 Follow this procedure to verify the Service Connection Point in Active Directory.

• Launch ADSI Edit as an Enterprise Administrator.
• Connect to the Configuration Naming Context of the domain.
• Browse to CN=Configuration,DC=contoso,DC=com > CN=Services > CN=Device Registration Configuration
• Verify that the leaf object CN=62a0ff2e-97b9-4513-943f-0d221bd30080 exists (this is the same CN value for every organization)
• Select Properties
• Select keywords from the Attribute Editor window and click Edit
• Verify the value of azureADId
• Verify the value of azureADName

This was all good for me.

OU Sync

The next step is to check that the device is in an OU which is synchronized to Azure AD. This is a configuration in Azure AD Connect. 

Navigate to the Domain and OU filtering screen and verify that the OU is selected for sync. I didn't find any problem here.

DSREGCMD

The next step was to have a look at one of the devices. You can get a lot of information using the DSREGCMD /STATUS cmdlet

In the Device State section I could see AzureADJoined = No. However the Diagnostics Data section gave me valuable information.

Failed to schedule Diagnostics Task. Error: 0x80041326

DSREGCMD /DEBUG /JOIN is also useful.

Cannot start Task: 0x80041326

Failed to schedule Join Task. Error: 0x80041326

This told me exactly where to find the problem. The issue was with the Workplace Join scheduled task.

Task Scheduler

Sure enough, I launched Task Scheduler and navigated to Microsoft > Windows > Workplace Join. I could see that all the tasks were disabled. In my experience some external configuration was preventing Azure AD registration on the device.

SCCM

ConfigMgr client settings was a good place to start. However automatic Azure AD registration was allowed (Cloud Services).

Group Policy

The next place to look was group policy, it's always a GPO, right?

A Resultant Set of Policy query showed me exactly where the problem was.

Computer Configuration > Administrative Templates > Windows Components > Device Registration > Register domain joined computers as devices was disabled.

What did this setting do? The description showed that this setting allowed devices to be silently registered in Azure AD. Disabling it obviously had the reverse effect and prevented the registration.

Solution

The solution was to remove this GPO setting from the affected devices.

I hope this helps. Until next time.....

Implementing hybrid AAD join over a VPN - the userCertificate attribute saga

Do you know what the userCertificate attribute is? If you don't then keep on reading.

This was fun. Once I learned more about how hybrid Azure AD join worked, I understood the behavior and was able to figure out how to make this work.

First, a little bit about the basics of HAADJ. What is a HAADJ device? This is a device that is joined to AD and registered with Azure AD. You can't be joined to both directories at the same time. Also you can't sign into a HAADJ device with Azure AD credentials. 

We know that we need an Azure AD Connect server. 

Version 1.1.819.0 and later of Azure AD Connect includes an option to configure Hybrid AAD Join. But what does that mean? When you choose this option the wizard creates a Service Connection Point in AD. The SCP contains all the information that a workstation needs to perform the HAADJ process.  

This isn't just a notional thing. You can actually see the SCP in AD using ADSIEDIT and connecting to the domain in the Configuration context. This is the SCP. The GUID (62a0ff2e-97b9-4513-943f-0d221bd30080) is what Windows 10 devices search for using an LDAP query, and is the same from organization to organization.

Have a look at the keywords and you will see the details of the specific organization's Azure AD domain and directory ID.

The next thing we do is to synchronize the workstations with AAD. Again we do this using Azure AD Connect. 

Check the box for the OU you need. The AAD Connect synchronization service runs on a schedule and creates computer objects in AAD for your workstations.

That's only the behind-the-scenes prep work though. On the workstation a scheduled task runs (Microsoft>Windows>WorkPlace Join). This task actually implements the Workplace join element. The device must have already synced to AAD for this to be successful.

That's it, that's all you need to do to implement HAADJ for an organization and to be fair, it just works. However there are some complications when most of the devices are remote and connect to the VPN infrequently. Then the process doesn't work and you need to understand the process and behavior a little more in order to make it work.

The first thing to understand is that the scheduled task really only runs when the user logs on to the workstation. That's the first trigger associated with the task.

There is a second trigger for this task. It runs whenever it detects Event ID 4096 in the event logs. In my experience though, this doesn't make any difference, and the task doesn't run again after the logon.

"Automatic device join pre-check tasks completed. The device can NOT be joined because a domain controller could not be located. The device must be connected to a network with connectivity to an Active Directory domain controller".

In the event log we get this continual error. We can see that workplace join fails as the device does not have line of sight to a domain controller. Unfortunately we are not connected to the VPN at that stage. The user may then connect to the VPN but at that stage it's too late, the scheduled task doesn't run again. You can see why this works seamlessly on the LAN, as the workstation is authenticating with a domain controller on log on and not using cached credentials.

OK, what if I run a script to run that scheduled task after the user connects to the VPN? I should be able to figure that out, right.

schtasks /Run /TN “Microsoft\Windows\Workplace Join\Automatic-Device-Join”

This is the command you need to run to kick that off.

However I found that it is not that simple. There are several steps involved to workplace join a device and you have to wait for the various schedules to see that in action. I'll explain the steps and introduce a new concept - the userCertificate attribute.

Azure AD Connect is configured to only synchronize devices that have the userCertificate attribute configured. AAD Connect does its job well and just filters out any devices that don't have this attribute set. You don't get an error or a warning to say that they are missing, which is a real nuisance.

So where does the attribute come from? These are two computers from the same organization, computer A has it and computer B doesn't. Computer A has synchronized with AAD but computer B has not. So how does that happen?

The science bit

Whenever the workplace join scheduled task runs it checks for a direct connection to a domain controller. When it finds one it looks for the Service Connection Point in AD using an LDAP query. The SCP has information on which AAD tenant the device should join. At that point the computer generates a self-signed certificate and the userCertificate attribute is added to the computer properties in AD. All is well at that stage and the computer will workplace join. Well, not really, quite a bit has to happen yet. If we run the workplace join scheduled task again at that point it will fail.

Automatic registration failed at join phase. Exit code: Unknown HResult Error code: 0x801c03f2. Server error: The device object by the given id (876df222-8565-4118-8193-f1d051d12e19) is not found.

It will tell us that the directory object cannot be found. Of course that makes sense. The device did not synchronize with AAD because of the missing userCertificate attribute. Now that we have forced that to happen, we must wait for the AAD Connect sync service to run again and this time the device sync successfully to AAD.

Now we must wait for the workplace join scheduled task to run again and this time, all the elements are in place and the workplace join is successful. This all works seamlessly for computers on the LAN but it does not work very well for VPN connected computers.

I created a task sequence that executes the workplace join scheduled task and I deployed it to VPN connected devices.

The script will execute the scheduled task every 10 mins. Then I encourage users to connect to the VPN and the magic happens.

• Script executes #1 (needs to have line of sight to DC) - scheduled task runs and finds SCP in AD with LDAP query. The computer generates a self-signed certificate and the userCertificate Attribute is updated in computer properties.
• AAD Connect sync service runs on a schedule - sync computer object to AAD
• Script executes #2 (needs to have line of sight to DC) - scheduled task runs and finds SCP in AD with LDAP query. Device is workplace joined.
• User that signs into the device will get an Azure AD user token that can be used to authenticate with Azure AD-based services

Don't forget to remove the deployment afterwards.

You can search AD to get a list of computers that are missing the userCertificate attribute.

Get-ADComputer -LDAPFilter '(!(userCertificate=*))' | Select Name,DNSHostName | export-csv -path C:\GH\Emptyattrib.csv

This will get you started but will include servers. You can filter them out.

You can also do a custom search on a specific OU using the LDAP query: 

(!(userCertificate=*))

I hope this helps you to understand what is going on with HAADJ. Until next time……..