We needed a recipe to try something new with Mahi Mahi we had in the freezer and wanted to try to use at least a few things from our recent CSA box as well.  We had fresh ginger in our box (first year of ginger for our CSA!) and some beautiful tat soi heads and these two recipes sounded like they’d go well together.    The Mahi Mahi recipe is compliments of everydaypaleo.com and the tat soi recipe is from our CSA newsletter (Andrea Yoder).  My wife did the cooking tonight and both dishes turned out excellent and the flavor combination was perfect!

Tat Soi Salad with Lemon-Ginger Vinaigrette (By: Andrea Yoder)

Ingredients:

• 1 head of tat soi, greens & stems in bite sized pieces, washed and dried
• 1 medium carrot, finely shredded
• Lemon zest from 2 lemons
• 2 Tbsp lemon juice
• 1 Tbsp honey
• 1 Tbsp fresh ginger, minced or pressed with a garlic press
• 4 Tbsp olive oil
• Pinch of salt
• Freshly ground black pepper
• 1/4 cpu almonds, chopped & toasted

Recipe:
Prepare the vinaigrette by combining the lemon zest, lemon juice, honey and ginger in a bowl.  Slowly drizzle in the oil and stir to combine.  Best if made 15-20 minutes before server to allow flavors to mix.

Prepare the salad just before serving by placing the tat soi and shredded carrots in a bowl and mix.  Top with pinch of salt and freshly ground black pepper.  Add the vinaigrette and toss to combine.  Don’t use too much vinaigrette, just enough to give a light coating.  Top with toasted almonds.

Gingered Carrots with Mahi Mahi (double this recipe if feeding more than 2 or 3)

Ingredients:

• 4 carrots, sliced
• 3 tablespoons grass fed butter or coconut oil
• 5 green onions, diced
• ½ teaspoon fresh grated ginger
• 1 teaspoon crushed garlic
• 1 pound of mahi mahi pieces or other white fish of your choice
• 1 tablespoon fresh lemon juice
• ½ cup chicken broth
• 1 tablespoon coconut aminos or 1/2 teaspoon of wheat free tamari
• Fresh ground black pepper and sea salt to taste

Recipe:

• Saute the carrots in 2 tablespoons of the butter or coconut oil over medium heat for 5-7 minutes or until the carrots start to brown.
• While the carrots are cooking, mix the chicken broth, coconut aminos or tamari and black pepper together in a bowl and set aside.
• Add the green onions to the carrots and cook for another minute.
• Add the lemon juice, ginger, and garlic to the pan and sauté just until the veggies are coated with the ginger and garlic.
• Sprinkle with a little sea salt, stir again, and remove the carrot mixture from the pan and set aside.
• Add the remaining tablespoon of butter or coconut oil to the same pan and melt over medium heat.
• Add the fish pieces and cook for 1 minute on each side, making sure that your pan is nice and hot so that the fish sear and turn golden brown on each side.
• Now, pour the chicken broth mixture that you made earlier over the fish, cover and cook for another 1 – 2 minutes or until the fish is tender and flakes apart easily.  Do not overcook!!
• Serve the fish over the gingered carrots

I had an interesting experience recovering a single node windows 2008 R2 cluster running multiple MSSQL 2008 instances.  We suffered a power failure that caused the server to reboot and after coming up the cluster service would crash at start.

Initially the only thing to go on was a single entry in the System Event Log for Event ID 1573:

Node ‘Servername’ failed to form a cluster.  This was because the witness was not accessible.  Please ensure that the witness resource is online.

I checked on the quorum disk and it’s there and marked as reserved as expected.  Head scratching commenced for a bit.  Tried a reboot just to make sure and had the same issue.  Tried to manually start the service and had the same issue.  Did some googling on the error and chased down a few items that ended up not being anything.

Tried to start the service with the fixquorum flag with no result.  Also tried to use the resetquorumlog with no result.

I discovered the cluster log command to generate a text file log of the cluster service which is when I finally started to make some progress:

Open a command prompt and run cluster log /g

This will output a file Cluster.log in C:\Windows\Cluster\Reports

On initial review of the log I found:

00000990.00000cf8::2012/11/14-13:18:25.643 ERR   mscs::QuorumAgent::FormLeaderWorker::operator (): ERROR_FILE_NOT_FOUND(2)’ because of ‘OpenSubKey failed.’

Which told me there was something wrong in the registry hive for the cluster.  The hive for the cluster is located in C:\Windows\Cluster and is a file called CLUSDB.  This file is automatically expanded and loaded under HKLM in the registry when the cluster service starts.  It was during this process that the server was crashing out so something was corrupted or wrong in the file.

My first attempt at a fix was to recover the CLUSDB file from a midnight snapshot taken about 3 hours prior to the power issue that caused the reboot.  Unfortunately this did not solve the problem which made me realize that something had changed or corrupted the file prior to the reboot and it just didn’t show itself until the reboot.  I went back to the Cluster.log file to see if I could find any more information.  I was regenerating the cluster.log file (cluster log /g) after each attempt to start the service to see if anything was changing and I notice something common with each startup:

000014b0.00000cf8::2012/11/14-13:25:39.708 DBG   [RCM] Resource ‘SQL Server (INSTANCENAME)’ is hosted in a separate monitor.
000014b0.00000cf8::2012/11/14-13:25:39.708 DBG   [RCM] rcm::RcmAgent::Unload()
000014b0.00000cf8::2012/11/14-13:25:39.708 INFO  Shutdown lock acquired, proceeding with shutdown

On each startup it would fail after the same INSTANCENAME and start to shutdown the service but I knew there should have been more Resources listed which meant the problem may be with the resource right after the last INSTANCENAME noted in the log.

With the cluster service stopped (so it wouldn’t try to restart and the hive wouldn’t be loaded) I launched regedit.  I navigated to HKLM and did a File->Load Hive and selected the CLUSDB file in C:\Windows\Cluster and gave it the name “Cluster” when prompted.  I then expanded the new cluster folder and then the resource folder and started to go through the list.  I quickly realized the order of resources in the folder matched how they were being noted in the Cluster.log file.  The resource that was next after the INSTANCENAME that was last noted in the Cluster.log was the Available Storage resource.  In looking at the keys for that resource I realized it had other resource ID’s listed in the “contains” key which should be storage resources that were in the Available Storage group except I knew that there shouldn’t be any.  I made note of the two resource ID’s in the contains key and went through the rest of the resources to make sure they didn’t actually exist and they didn’t.  I then went back to the contains key for the Available Storage resource and edited it and removed the two entries.  I then highlighted the Cluster folder under HKLM and unloaded the hive File->Unload Hive and then closed out regedit.  I started up the cluster service manually and this time everything started up correctly.

So what happened?

Roughly 2 weeks prior to this outage an Instance had been removed from the cluster.  It had 4 storage devices associated with it which were initially moved to the available storage group after being removed from the instance group and then were deleted as disks from the cluster.  Apparently this process (done via the failover cluster gui) failed to fully remove 2 of the 4 objects from the registry correctly.  I’ve found a few other people suggesting to always use the command line cluster program to remove resources to be extra safe which I plan to do from now on.  The problem did not show up until the next time the cluster service restarted.