Orchestrator 2012 Web Service Request Issue

I've started running into an issue w/ external requests to start a runbook via the web service.  I'm interested to see if others are experiencing the same issue.

It seems sporadically that requests coming in (w/ or w/o parameters) will fail and the web service will return a 405 (Method Not Allowed) status code with the response.  Trying the same request again moments later would succeed.

Turning on Failed Request Tracing (FRT) in IIS for the "Microsoft System Center 2012 Orchestrator Web Service" will reveal more details w/ the error. 

From the site highlighted in IIS, you can enable FRT from the Actions pane.  After FRT is enabled, you can create the rule to capture specific status codes (400,404,405,500 in the example below).







After the issue occurs and the request fails w/ a code of 405, a log file will be generated in the following folder if you accepted the default path - C:\inetpub\logs\FailedReqLogFiles\W3SVC2.  Towards the bottom of the log file, you'll find this error information:

<Data Name="Buffer">&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot; standalone=&quot;yes&quot;?&gt;
&lt;error xmlns=&quot;http://schemas.microsoft.com/ado/2007/08/dataservices/metadata&quot;&gt;
  &lt;code&gt;&lt;/code&gt;
  &lt;message xml:lang=&quot;en-US&quot;&gt;The requested operation requires Publish permissions on the Runbook&lt;/message&gt;
&lt;/error&gt;</Data>

This appears to be a bogus error since sending the same request again will succeed using the same credentials.

After a little more digging, the error occurring seems to correlate w/ the ClearAuthorizationCache maintenance task in the SQL database.  Since this task computes/populates the folders, runbooks, and permissions....it theoretically make sense that while that task is running, incoming requests would fail since it thinks the user does not have permissions to the runbook (when it does!).  This could technically also affect requests being sent through the Orchestration Console to stop/start jobs.  By default, this task runs every 600 seconds (10 minutes).  I'm not sure of the duration it takes for the ClearAuthorizationCache task to complete, but it would depend on how much data there is to process (# of runbooks, folders, etc. in the environment).


If you have seen or do experience similar issues, I'd appreciate if you left a comment.  I also suggest opening a case w/ Microsoft to determine if this is a general product issue.

Orchestrator 2012 - Daylight Savings Time Scheduling Issue

I'm running a little behind on this post as Daylight Savings Time here in the US was back on Sunday, November 4.  For ease of example, all dates/times will reference the schedule for Daylight Savings in the United States.

There is an issue with scheduling Runbooks that start with the Monitor Date/Time activity during DST.

Daylight Saving Time (United States) 2012 began at 2:00 AM on Sunday, March 11.  At this time, 
the clocks would be moved ahead one hour to 3:00 AM.

• This means any Runbooks scheduled to run between the hours of  2:00 AM - 3:00 AM would be skipped entirely.

Daylight Saving Time (United States) 2012 ended at 2:00 AM on Sunday, November 4.  At this time, 
the clocks would be moved back one hour to 1:00 AM.

• This means any Runbooks schedule to run between the hours of 1:00 AM - 2:00 AM would run twice.

Depending on what the runbooks are executing, this could cause major issues and/or unscheduled downtime. It would also suggest Runbooks that are not set on an hourly interval (i.e. once daily) be scheduled outside of the hours of 1:00 AM - 3:00 AM to avoid any interruptions.  Unless you throw an exception into the schedule to work around the timing that the clocks are changed.

Dynamic Sleep Activity to Schedule Runbook

I was designing a workflow where there was a Main Runbook that called RunbookA and Runbook B. 

The catch was RunbookA was invoked at 3pm and looped for 3 hours until a file was processed successfully (so there was no standard end time).  Also, RunbookB needed to run at 6pm only if RunbookA was successful.

So I needed a way to determine/schedule RunbookB to run at 6pm if RunbookA completed at 3pm or 4pm or 5pm (you get the point).  So I used PowerShell to create a dynamic Sleep based on when RunbookA completed to invoke RunbookB at 6pm.

High Availability for Runbook Servers on Invoking Runbooks

Here's just a quick tidbit for specifying runbook servers used in the Invoke Runbook standard activity.  It’s beneficial to keep the runbook server names stored in Global Variables.  This way you can insert the variable in the Invoke Runbook activity instead of hardcoding the server name.

In case a runbook server has issues and is powered down or unavailable, the invoke runbook activity will automatically start the invoked runbook on the next runbook server in line.  Another benefit is the ability to add/remove runbook servers and just update a variable than trying to find/replace every invoke runbook activity w/ the new server name.

You can also use the variables to group runbook servers based on their role in the environment.  So to load balance runbooks, depending on how many runbook servers you have (for this example we’ll say three internal).  You could specify three variables which have the following values.

Primary:  myrunbookserver1;myrunbookserver2;myrunbookserver3
Secondary:  myrunbookserver2;myrunbookserver3;myrunbookserver1
Tertiary:  myrunbookserver3;myrunbookserver1;myrunbookserver2

So for runbook servers interacting w/ servers on an internal domain you could use:



So for runbook servers interacting w/ servers on a dmz domain you could use:



Finally, subscribe to the variable in the Invoke Runbook activity.

Start Runbooks from SharePoint List

After stopping all running/pending runbooks to perform maintenance on the SCOrch environment, it becomes cumbersome to start all runbooks that begin with a monitor date/time activity (especially when the number of runbooks to start climbs above 100….).

You can create a SharePoint list (Figure A) of runbooks that need to be started after maintenance (good to document nonetheless) that Orchestrator can reference to start all of them automatically!

The main field that we need to use is the RunbookID in the SP list.  I also use a field called "Active" in the list to exclude workflows that may not need started in case there is any reason that they shouldn't be started for the time being.

Figure A

The workflow is made up of three runbooks.  A main which queries the SP list for the active runbook ids to start, the execute which starts the targeted runbook, and a seperate runbook to keep track of processed runbooks to later check for any errors.

Main (Figure B)

Figure B

The main starts by resetting a counter to cleanup from previous runs.  Then it uses the Microsoft SharePoint IP by Jeff Fanjoy (more info here http://orchestrator.codeplex.com/releases/view/75877) to query the list identified in the SP IP configuration in options.  Within the properties of the Get Active Runbooks activity, the filter is modified to choose the "Active" field set on the list object (Figure C).

Figure C

From the runbooks found in the list query, it will invoke the execute runbook with the runbook id and runbook name.  It also invokes the counter modification runbook with a parameter – increment (note this invoke must set the property to wait for completion – See Figure E).

Exploring the execute runbook that's invoked in Figure D….it uses the Standard Logging IP (more info here http://orchestrator.codeplex.com/releases/view/76097).  It starts the targeted runbook (you can reference this from my previous post – http://jmattivi.blogspot.com/2012/06/scorch-powershell-to-start-runbook-part.html).  Upon success or failure to start the runbook, it invokes the counter modification runbook with a parameter – decrement (note this invoke must set the property to wait for completion – See Figure E).

Figure D
Figure E

Moving back to the Main runbook (Figure B), it waits for the counter value to be 0 – indicating processing has completed.  If not it hits a timeout after 15 minutes and sends an error email for investigation.  It then queries the Standard Logging table in the database for any failures to start runbooks from the execute runbook.  If errors are found, it uses a Powershell script to parse the flattened data back into multi-valued data and send that information in an email.  If no failures are found, it sends a success email that all targeted scheduled runbooks have been started successfully.

SCOrch – Powershell to start Runbook – Part 2

Here is an example to start a Runbook w/ parameters from Powershell.  It’s pretty much the same exact as a standard POST to start a Runbook, but also includes the parameters in the properties struct.

I’ve found that the free program Fiddler helps to find the guids and struct needed to build the POST.  You can also use the Orchestration Console to drill down through to the Instance Summary and Activity Details to find the guids for each parameter.

This is the parameters struct that is added to the regular POST.

Parameter1/Parameter2 are the names that you give the parameter.

You need to specify the guid for each parameter.

Value1/Value2 are the input values you’d like to specify for the Initialize Data activity.

<d:Parameters>&lt;Data&gt;&lt;Parameter&gt;&lt;Name&gt;Parameter1&lt;/Name&gt;&lt;ID&gt;{10000000-0000-0000-0000-000000000000}&lt;/ID&gt;&lt;Value&gt;Value1&lt;/Value&gt;&lt;/Parameter&gt;&lt;Parameter&gt;&lt;Name&gt;Parameter2&lt;/Name&gt;&lt;ID&gt;{20000000-0000-0000-0000-000000000000}&lt;/ID&gt;&lt;Value&gt;Value2&lt;/Value&gt;&lt;/Parameter&gt;&lt;/Data&gt;</d:Parameters>

Here is the complete script to include the parameters in the POST to start the Runbook.  The highlighted fields would need updated to your environment.

 ##############################################################################

$creds = Get-Credential("domain\username")

$url = "http://scorch.domain:81/Orchestrator2012/Orchestrator.svc/Jobs/"

$request = [System.Net.HttpWebRequest]::Create($url)

$request.Credentials = $creds

$request.Timeout = 60000

$request.Accept = "application/atom+xml,application/xml"

$request.Headers.Add("Accept-Charset", "UTF-8")

$request.ContentType = "application/atom+xml"

$request.Method = "POST"

$requestBody = '<?xml version="1.0" encoding="utf-8" standalone="yes"?>

<entry xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns="http://www.w3.org/2005/Atom">

<content type="application/xml">

<m:properties>

<d:Parameters>&lt;Data&gt;&lt;Parameter&gt;&lt;Name&gt;Parameter1&lt;/Name&gt;&lt;ID&gt;{10000000-0000-0000-0000-000000000000}&lt;/ID&gt;&lt;Value&gt;Value1&lt;/Value&gt;&lt;/Parameter&gt;&lt;Parameter&gt;&lt;Name&gt;Parameter2&lt;/Name&gt;&lt;ID&gt;{20000000-0000-0000-0000-000000000000}&lt;/ID&gt;&lt;Value&gt;Value2&lt;/Value&gt;&lt;/Parameter&gt;&lt;/Data&gt;</d:Parameters>

<d:RunbookId type="Edm.Guid">00000000-0000-0000-0000-000000000000</d:RunbookId>

</m:properties>

</content>

</entry>

'

$requeststream=new-object System.IO.StreamWriter $request.GetRequestStream()

$requeststream.Write($requestBody)

$requeststream.Flush()

$requeststream.Close()

$response=$request.GetResponse()

$requestStream=$response.GetResponseStream()

$readStream=new-object System.IO.StreamReader $requestStream

$Output=$readStream.ReadToEnd()

$readStream.Close()

$response.Close()

##############################################################################

SCOrch – Powershell to start Runbook – Part 1

I have seen a few examples of ways to start a runbook with C and VB.  However, I haven’t come across any ways of using Powershell to start a runbook successfully.

This will be the first post of a series of different ways to interact w/ the web service through Powershell.  I’ll also post how to start w/ parameters, stop a runbook, get all running/pending runbooks, etc….

Here’s the short of it.  I Haven’t built any logic, parameters, or error handling into it yet….just wanted to get the POST to work with Orchestrator’s web service.

You would need to update the credentials, url for the web service pointed to the Jobs collection, and the RunbookID in the request body (highlighted below).

##############################################################################
$creds = Get-Credential("domain\username")
$Output = $null
$Success = $null
$url = "http://scorch.domain:81/Orchestrator2012/Orchestrator.svc/Jobs/"
$request = [System.Net.HttpWebRequest]::Create($url)
$request.Credentials = $creds
$request.Timeout = 36000
$request.Accept = "application/atom+xml,application/xml"
$request.Headers.Add("Accept-Charset", "UTF-8")
$request.ContentType = "application/atom+xml"
$request.Method = "POST"

#Request w/o Parameters
$requestBody = '<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<entry xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns="http://www.w3.org/2005/Atom">
<content type="application/xml">
<m:properties>
<d:RunbookId type="Edm.Guid">RunbookGUID</d:RunbookId>
</m:properties>
</content>
</entry>
'

$requeststream=new-object System.IO.StreamWriter $request.GetRequestStream()
$requeststream.Write($requestBody)
$requeststream.Flush()
$requeststream.Close()
$response=$request.GetResponse()
$requestStream=$response.GetResponseStream()
$readStream=new-object System.IO.StreamReader $requestStream
$Output=$readStream.ReadToEnd()
$readStream.Close()
$response.Close()

$Output
##############################################################################