iSiek's forum about Microsoft Windows services

Full Version: DC Crash in 2 DC Domain - How to fix?
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hello Krzysztof,

I am pulling my hair for the past weeks trying to fix a tiny (2 DC) network/domain problem. The DC in my server room crashed - and all I have is App and User data that we back up weekly...

The domain is very small, and shared across 2 locations with a single DC and 15 users at each. One is here in my building, and the other about 15Km away across town. Both are Msoft 2008 R2 Server Std-based DC.

OK, so they are essentially clones of each other, replicating and sharing data across a secure link and mine crashed - the OS/boot HDD died, won't spin up and it won't rebuild either, so I thought I'd save alot of time by simply taking a bit-copy image of the 3 partitions of the server across town, bring it back here and making a clone of it on a freshly installed HDD on this box. The crashed DC had 2x HDD, 1 a 250Gb with 2 partitions (OS Binaries and App Data) and the other a 160Gb with all User Data.

OK, so I used all the tools, tricks and skills I learned, Using a LINUX boot cd on the good DC and successfully created a clone of both of it's HDD. After bringing it back online and verifying operation, I came back here and imaged a new HDD for my DC, cracked the no-boot issues I encountered and am finally IN :-)

As you have probably guessed, my big problem is that now MY DC has the same DC, IP and Machine name as the other one so I cannot simply 'rejoin the domain' but have to fix that first. What is the best way?

I have not tried any DCPROMO stuff yet from your blog, but that info did not address a crashed DC that MUST be reinstalled and made operational, up and running. - I can rebuild the Root/Boot OS Binaries partition easily as I now have a bootable GHOST of it. (~2hrs time for that)

Any help appreciated. A friend told me I should have just installed fresh 2008 R2 Std, made it a DC, rejoin the domain and then set up all the replication again - but I have not 'built' a DC since 1992 :-) and in my job I am never able to spend any more than 15 min on a task and this will certainly be one of those. I made the copy of the good DC on a Saturday, and now I am ready to fix this today (also a Saturday)

Your blog is the most clear, concise and informative I have ever seen, and you must be a proud MCSE to know your stuff so well in order to freely share it with we that so desperately need it :-)

Cisco
Hello Steve,

I'm glad you wrote to me about this issue. I will try to help you as best as I can.

Thank you for detailed description of your environment and the issue, that helps a lot!

Quote:Any help appreciated. A friend told me I should have just installed fresh 2008 R2 Std, made it a DC, rejoin the domain and then set up all the replication again

Your friend is right! You should install fresh copy of Windows Server 2008 and promote it as Domain Controller. In other case when you colne the existing one, you may fall in troubles and lingering objects may appear in your domain.

My friends, Wojciech and Sandesh have a good entry for that on their blogs:

Domain Controllers cloning was introduced for the first time with Windows Server 2012 and it has some additional requirements, the most importnat is hypervisor supporting VM Generation ID feature. In other case, you would not be able to clone DCs.

But... first things, first.

  1. Step 1
    Your old DC is broken and cannot be brought up again. That means, it was not decommissioned properly and some metadata left in your AD environment. The other DC still "knows" the broken DC and tries to "talk" (replicate objects). You need to clean up AD by removing metadata of the broken Domain Controller. Its objects till should exist under "Domain Controllers" Organizational Until while browsing Active Directory Users and COmputers console. If so, just follow steps described in this artilce on my blog at http://kpytko.pl/active-directory-domain...-over-gui/

    If this would not help, you may try to follow article showing how to do that over ntdsutil in command prompt at http://kpytko.pl/active-directory-domain...ontroller/

    When you finish cleaning up you environment go to step no. 2

  2. Step 2
    Install fresh Windows Server 2008 server or higher if you have licenses. This should not be a problem, but you may watch a video on my youtube channel showing how to do that with Windows Server 2012R2. The process is really similar for Windows Server 2008, so it is available at http://kpytko.pl/windows-server-2012/ins...2r2-video/

    Please follow also post installation steps video to see what it's required to start promoting new Domain Controller.

    I'm really sorry for missing voice and comments but had no possibility to do that :/

  3. Step 3
    Just verify if there are no serious errors in your environment before you will start promoting new Domain Controller. Some tools with examples are avilable at http://kpytko.pl/active-directory-domain...ing-tools/

  4. Step 4
    Now, when you have cleaned up Active Directory environment and installed new server for Domain Controller, just follow steps described in my article at http://kpytko.pl/active-directory-domain...ontroller/ The example shows how to do that for Windows Server 2008R2 but this is EXACTLY the same procedure for Windows Server 2008

When you follow all of the above guides, you should have successfully built additional Domain Controller. Just wait a while to allow AD database and SYSVOL replication. And voila! You're done!

I hope this was helpful for you! Please let me kow after all!
Well, I did not get your reply this am when I went to shop, and didn't want to waste a day of trying either so here is what I did.. and got nowhere...

First I booted into local admin mode on the cloned DC and ran DCPROMO /forceremove which took 30 minutes and turned the/my 'clone' of the other DC back into a plain old server. Then I rebooted and launched Computer>Properties and changed the name and IP, GW, Mask and 'swapped positions of the DNS entries so they too were back where to the old original DC settings were.

Next, I rebooted, crossed my fingers and tried to run DCPROMO... AND it failed saying a DC with that name already existed in the domain (of course It did) so I opened an RDP session to the remote DC, launched Server Manager on it, and went thru all menu options trying to RESET, RENAME or something, but all it would let me do was REMOVE (Rt Click and Delete) my old DC. So I did. I looked, but there was no option to rename this type object - I looked everywhere...

It complained about Global Catalogs, DNS and FSMOs-something but as the original DC HDD was toast and I had no other servers or DC in the domain, all I could do was keep clicking NEXT.. Now that the pesky old DC wasn't there anymore, I minimized the RDP session and tried DCPROMO again.

Crap - Now it could not find the specified domain that I'd just used 5 min ago (?) Wait.. it just DID find the specified domain b/c it warned me that it could not promote my new box b/c the (old) name still/already existed.... Hmmm.

Well, I was sitting there for a long time thinking WHAT HAVE I DONE, and then it occurred to me that it may be the remnants of the old DC in there (that I read about on the web) casing the stoppage. I let it sit for 1/2 hr while I answered emails and did inventory, but when I tried it again, I got the same response.. so now I thought, OK, how about if I rename my box to something that never existed before, which might get thru b/c there would be no remnants and there would be no name/DNS issues - so I did, then rebooted and tried again. No go joe! (snif) This time it reported the there was no DNS registry for the name I used for my DC, and I have no idea how to generate one, nor do I know anything about regenerating global catalogs or performing FSMO seizures

Well now it was getting into the 6th (!) hour without lunch or a break and I thought OK, I'll try one more thing and then I have to get out of there or I would do something really bad, so I tried rejoining that plain old server to the old domain.. and that failed. (sigh) Knowing I could always reimage the boot/root from the Acronis Image I grabbed 2 weeks ago made it less painful, but NOTHING was working so I went back to read some more web tips on crashed servers and Domain Controller building.

I read a bit about Ntdsutil.exe which is on my machine and does not work from the Admin-level CMD prompt. If that is one of the tools that DCPROMO 'calls' it definitely could be at least one plug in the works. The last thing I did was RDP to the Remote box and create an 'empty' appropriately-named DC, but all I could do was pre-stage a Read-Only DC - and I didn't go thru with that and left to get some supper. Frustrating!

Tomorrow I'll try DCPROMO again, and if it fails, I'll roll back to reimaging the Boot/Root and starting over.
OK, please let us know about your current status. If you were able to solve the issue or you still need a help.

We can try to start solving that problem step by step but we need to know where we are, now.

One question to you.
Do you have latest Domain Controller system state backup or any valid (up to 180 days)?
If so, in the worst case, we can try with authoritative DC restoration instead of restoring the image.

Oh and one more.
Do you have possibility to use virtual machine on any hypervisor? We can try to bring your domain up on the VM to speed up tests and troubleshooting.
Krzysztof,

Only App and User Data backups avail, nothing set up by my predecessors to run full or bare-metal backups or even a Ghosted image. Sorry.

I didn't go in today as I had other stuff to do, but might go in tomorrow and take another whack at it. I will Google Authoritative Restoration and see what I find - and no... we have no accessible virtual environments, but I might be able to repurpose ome old hardware and make one.
S.
Quote:Only App and User Data backups avail, nothing set up by my predecessors to run full or bare-metal backups or even a Ghosted image.  Sorry.

Not good, not good as Domain Controller system state backup is the only one allowed in this case to rebuild DC or domain.

Please tell me one more thing. You have 2 DCs for your AD. One of them is broken in remote location. The second is still working fine? Which of them held FSMO roles? You can check that by typing in command prompt

Code:
netdom query fsmo

If we have at least one good DC we would be able to simply bring the second up.

At this stage where we try to repair your environment, you can use for virtualization even VirtualBox, Hyper-V client or VMWare Workstation to speed up the process. Of course later in production they should not be used at all. We will migrate virtual DC to the physical box and decommission VM. But this is much more convenient to use VM instead or rebuilding hardware OS every time.

OK, please update us with your latest progress. Thank you in advance.
(16-02-2015, 12:41 PM)iSiek Wrote: [ -> ]
Quote:Only App and User Data backups avail, nothing set up by my predecessors to run full or bare-metal backups or even a Ghosted image.  Sorry.

Not good, not good as Domain Controller system state backup is the only one allowed in this case to rebuild DC or domain.

Please tell me one more thing. You have 2 DCs for your AD. One of them is broken in remote location. The second is still working fine? Which of them held FSMO roles? You can check that by typing in command prompt


Code:
netdom query fsmo

If we have at least one good DC we would be able to simply bring the second up.

At this stage where we try to repair your environment, you can use for virtualization even VirtualBox, Hyper-V client or VMWare Workstation to speed up the process. Of course later in production they should not be used at all. We will migrate virtual DC to the physical box and decommission VM. But this is much more convenient to use VM instead or rebuilding hardware OS every time.

OK, please update us with your latest progress. Thank you in advance.
Hello Steve,

what about progress in your case? Were you able to fix the issue?