[Part-5]Private Cloud: Some common issues and troubleshooting tips

This is the fifth and final post in the series of blog posts on how to build own Private Cloud using Windows Azure Pack (WAP).

Here is a list of topics for the entire series:

[Part-1]Private Cloud: Introduction and installation of pre-requisites

[Part-2]Private Cloud: Configure SCVMM and SPF

[Part-3]Private Cloud: Install and Configure WAP

[Part-4]Private Cloud: Demo of provisioning VM and SQL DB using Tenant portal.

[Part-5]Private Cloud: Some common issues and troubleshooting tips.

In this blog I will provide some troubleshooting tips e.g. where to look in case of any error and few scenarios.

When it comes to troubleshooting, usually, the following questions comes into mind:

Log files: Where and what to look at?
Troubleshooting example
Common issues/errors

I am going to cover the above mentioned topics in this blog post.

1. Log files: Where and what to look at?

It is important to know that every activity in WAP will create a corresponding job in SCVMM. So, the first place to look for is the related history of the job in SCVMM.

Go to SCVMM console > Jobs to check the job history.

If nothing is evident from the job history, likely the error occurred at a prior stage reaching to SCVMM. Look at the Event Viewer logs in this case.
Go to Event Viewer > Applications and Services Logs > Microsoft

Look under the folder/component possibly raising the error. e.g. WindowsAzurePack

If you want to dig deeper, enable debug tools called Developer Tool. Developer Tool is under Settings within Internet Explorer (IE 10).

You need to Refresh the page to find the tracing tool at the bottom in IE 10.

Now, you can co-relate things with the other log files e.g. HTTPLOG.

To see HTTPLOGs, go to Run prompt, type logfiles and open HTTPERR

2. Troubleshooting example

Let us see one of the scenarios I faced and the troubleshooting steps to reoslve.

ISSUE:

I was getting following error when I was logging into Admin portal with a red icon.

As a first troubleshooting step, I looked at the SCVMM job history, nothing obvious there. It was likely because error occurred while loading the page indicating it has occurred prior to VMM.

As a next step, I went to Event Viewer > Application and Services Logs > Microsoft > WindowsAzurePack > MgmtSvc-AdminAPI

Not conclusive information in Event logs too.

Next step – I started the Developer Tools and looked into the Network Profile.

As you see above, error is pretty clear – HTTPS Error 400 means Bad Request

Carefully reviewing event logs again, I got more clue. It seems to be related with user authentication token.

+++++++++++++++++++++++++++++++++++++++++++

&lt;HttpContext&gt;

    &lt;User IsAuthenticated="true" Name="Manish\administrator" /&gt;

    &lt;Request&gt;

      &lt;RawUrl&gt;/Plan/List?ts=1423229252677&lt;/RawUrl&gt;

      &lt;UserHostAddress&gt;10.52.18.171&lt;/UserHostAddress&gt;

      &lt;Headers&gt;

        &lt;Header Key="Cache-Control" Value="no-cache" /&gt;

        &lt;Header Key="Connection" Value="Keep-Alive" /&gt;

        &lt;Header Key="Content-Length" Value="56" /&gt;

        &lt;Header Key="Content-Type" Value="application/json"

        &lt;Header Key="x-ms-client-antiforgery-id" Value="ws3O12S2LNeg1msZCk1y0nO+LVQ8W9dOzX7n7RV0CFkwWoOnDx/xRrylvnxRsKeW2KL+hCIK0RkaT+ZAAFli+EdkqPiAzdQOLsGSpUnZh6BW8NO3SLTS/cg1Gd8IA0C9wXvEOeBzKEaeH1o1G5CpFw==" /&gt;

        &lt;Header Key="x-ms-client-request-id" Value="63ef3416-4e64-450e-87b2-559a05fe851a-2015-02-06 13:27:32Z" /&gt;

        &lt;Header Key="X-Requested-With" Value="XMLHttpRequest" /&gt;

        &lt;Cookies&gt;

          &lt;Cookie Name="sub" Secure="false" Expires="0001-01-01T00:00:00Z" Domain="" Path="/" Value="Redacted (4 characters)" /&gt;

          &lt;Cookie Name="AdminSiteFedAuth" Secure="false" Expires="0001-01-01T00:00:00Z" Domain="" Path="/" Value="Redacted (2000 characters)" /&gt;

          &lt;Cookie Name="AdminSiteFedAuth1" Secure="false" Expires="0001-01-01T00:00:00Z" Domain="" Path="/" Value="Redacted (2000 characters)" /&gt;

          &lt;Cookie Name="AdminSiteFedAuth2" Secure="false" Expires="0001-01-01T00:00:00Z" Domain="" Path="/" Value="Redacted (2000 characters)" /&gt;

          &lt;Cookie Name="AdminSiteFedAuth3" Secure="false" Expires="0001-01-01T00:00:00Z" Domain="" Path="/" Value="Redacted (716 characters)" /&gt;

          &lt;Cookie Name="__RequestVerificationToken_Admin" Secure="false" Expires="0001-01-01T00:00:00Z" Domain="" Path="/" Value="Redacted (128 characters)" /&gt

++++++++++++++++++++++++++++++++++++++++

HTTPLOG confirms this:

It is now evident that their is a problem with the request being passed.Further research led me to this KB article which explains the root cause as authentication token length longer that allowed.

http://support.microsoft.com/kb/820129

I changed the user account to local Administrator and issue resolved.

So, looking at the logs mentioned in first section, you should be able to resolve most of the common issues.

3. Common issues/errors

1. You may get following error when you try to create a VM from WAP.

Failed to create virtual machine 'vmname'.
Failed to submit operation request.

Check the event log to find these errors:

Log Name: Microsoft-WindowsAzurePack-MgmtSvc-TenantSite/Operational

 Source: Microsoft-WindowsAzurePack-MgmtSvc-TenantSite
 Date: 1/8/2015 3:16:51 AM
 Event ID: 232
 Task Category: (59)
 Level: Error
 Keywords: None
 User: IIS APPPOOL\MgmtSvc-TenantSite
 Computer: myvmserver.manish.com
 Description:
 Potential version mismatch between WAP and SPF, please verify both component's versions and if they're compatible, Exception: 'System.Data.Services.Client.DataServiceRequestException: An error occurred while processing this request. ---> System.Data.Services.Client.DataServiceClientException: <?xml version="1.0" encoding="utf-8"?><m:error xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"><m:code /><m:message xml:lang="en-US">An error occurred while processing this request.</m:message><m:innererror><m:message>The property 'DynamicMemoryMinimumMB' does not exist on type 'VMM.VirtualMachine'. Make sure to only use property names that are defined by the type.</m:message><m:type>Microsoft.Data.OData.ODataException</m:type><m:stacktrace> at Microsoft.Data.OData.ReaderValidationUtils.ValidateValuePropertyDefined(String propertyName, IEdmStructuredType owningStructuredType, ODataMessageReaderSettings messageReaderSettings, Boolean& ignoreProperty)

Resolution:

Follow this KB article if you are hitting the same issue.

http://support.microsoft.com/kb/2983453

If the KB article doesn’t help, it might be an issue with version mismatch. Please make sure all your component SCVMM, SPF, WAP are on the latest build.

2. When you try to create a VM you may receive following error.

The server encountered an error while creating “Name of the VM”

When you check the VMM log you will find errors mentioned below:

Error (12700)

VMM cannot complete the host operation on the myvmserver.manish.com server because of the error: The operation failed.

User “ Domain\manish” failed to create external configuration store at 'C:\ProgramData\Microsoft\Windows\Hyper-V\Test': A required privilege is not held by the client. (Virtual machine ID 0x80070522)

Unknown error (0x8000)

Recommended Action

Resolve the host issue and then try the operation again.

Cause:

By default the local User Right “Create Symbolic links” is assigned to the Adminstrators group. Once you install Hyper-V a extension is installed on this machine that allows Service SIDs to be used and the Service SID “Virtual Machines” is allowed to “Create Symbolic links”

Resolution:

To get the necessary Service SID added, you need to edit the GptTmpl.inf that is associated with the GPO

Example with Administrators and added “Virtual Machines”

….
 [Privilege Rights]
 SeCreateSymbolicLinkPrivilege = *S-1-5-32-544, NT Virtual Machine\Virtual Machinesn the Sysvol directly. [You need to locate the correct GptTmp.inf by the GPO's Unique ID]

And, that concludes this series of Building Your Own Private Cloud using Windows Azure Pack.

Thank you all for following the series and our blog.

Stay tuned for some more interesting blogs.