I wanted to shoot you a quick note because I'm hoping you can help me. I work for a company called Windward IT Solutions and we specialize in implementing SCORCH along with a few other workflow development tools. Right now we are in desperate need of another automation expert. I came across your blog and it evident that you have a lot of expertise with SCORCH and runbook automation.
I was hoping you might be able to point me in the direction of a colleague that might be interested in the position and learning some of the other automation tools out there.
I'd appreciate any help or suggestions you can provide as to how I can get in touch with some people that have experience with SCORCH. If this is something you're interested in as well I'd be happy to speak with you about the opportunity.
Thanks,
Chris Werner Technical Recruiter Windward IT Solutions 703 812 0117
We are using SCORCH 2012 R2. We still see some orphaned jobs on the Monitor runbooks. Is there any ways you know to fix it? We have already implemented the steps in this link http://www.concurrency.com/blog/how-to-clear-orphaned-orchestrator-runbook-instances/
Can you confirm that you've actually stopped the running monitor runbooks with the orphaned jobs? The stored proc will only removed the orphaned instances if the runbook is NOT running. I don't think I've encountered a situation where the runbook has been stopped and the stored proc doesn't remove them.
I've used these queries back in the Opalis days to clear up the orphaned instances (update statements being commented out).
****Find and remove orphaned policies**** select * from dbo.policyinstances where timeended is null and status is null
--update dbo.policyinstances --set [timeended] = getdate() --set [status] = 'warning' --where timeended is null and status is null
****Find and remove orphaned runbooks for specific runbook**** select POLICYINSTANCES.UniqueID, POLICIES.Name from dbo.policyinstances, dbo.POLICIES where status is null and (POLICYINSTANCES.PolicyID = POLICIES.UniqueID) and POLICIES.Name = '' Order By PolicyID
--update policyinstances --set policyinstances.[timeended] = getdate() -- ,policyinstances.[status] = 'warning' --From POLICYINSTANCES, POLICIES --where policyinstances.status is null and (POLICYINSTANCES.PolicyID = POLICIES.UniqueID) and POLICIES.Name = ''
On a side note....You don't necessarily have to go into SQL to run the stored proc to clear orphaned instances. Right-clicking the management server in the Designer console and selecting "Log Purge" and Purge Now will also run that stored proc to remove the orphaned instances.
Yes we had to stop runbooks and clear orphans. But is there a way to get permanent fix for this, does microsoft have a solution to avoid orphans? It is creasing our runbook servers sometimes, our runbooks stop working completely. Any ideas to fix it?
Orphaned jobs are created (most of the time....) when runbook servers have connectivity issues with the database. The runbook server will go into a sleep mode for three minutes wherein all running runbooks will cease to run on that particular server and start running on another runbook server. After three minutes, the runbook server will continue to accept requests to run jobs.
Orphaned instances don't really cause any performance issues (to my knowledge) other than being annoying while still visible.
I don't foresee Microsoft changing any code to prevent the orphaned instances altogether....especially when the stored procedure is in the database to clear orphaned instances once the runbook is stopped.
Hey Im back again. same person.. can you reply to the above comment if possible. http://jmattivi.blogspot.com/2012/06/high-availability-for-runbook-servers.html
you have mentioned here the global variables have high availability. We have situation in SCORCH R2, where the jobs pile up. We have 8 runbooks servers. Each runbook server can take 100 jobs max load. When 3 runbook servers hit their max threshold, the performance of SCORCH becomes slow. hear teams mailing us alerts are not processed. can you help in this case?
It's never a good thing when runbook servers are hitting their max limit.
In this case, the solution would be to either
Change the target runbook server (other than one of the three) for a bunch of the runbooks to spread the load out that they're not getting maxed out. OR Increase the max jobs for the runbook servers to something acceptable for your needs (above 100).
Does the passphrase contain any special characters? I ran into that issue a while ago that produced a similar error, I believe, was classified as a bug.
You could try creating a test set of keys to decrypt and create a passphrase with uppper and lower case characters (I think numbers were fine as well) to test. But I would guess that the "*" is creating the problem.
I really would rather not pursue that since it would involve contacting each vendor we receive files from and the work to do that would be a major pain. Do you know of any other custom decrypt integration packs that I can try doing this with?
Technically, you would need to generate new keys and just provide the vendor the new public key to use going forward. The new passphrase would then be updated in the runbooks.
Otherwise, I believe the only option would be to use the Run Program activity with gpg from the command line.
I figured out how to get this to work. I isolated our secure and public ring files in one directory and pointed the decrypt activity keyring folder setting to that directory and that did the trick. Apparently opalis is smart enough to pair the appropriate files together to decrypt files but orchestrator isn't.
Awesome! Thanks for posting back! I can't imagine there were any changes to the PGP activities between Opalis and Orchestrator....but at least there's a workaround.
I have 4 runbooks designed to handle quarterly audits, however I would like them to kick off automatically. I've tried using the monitor/check schedule, but to only start quarterly means I’m manually adding in exceptions (I’m trying to manual maintenance). One idea I had was to have my check schedule set on one day per month, then add an if statement in my first .Net script object (using Powershell). So if the current month variable doesn’t match the audit months, the whole runbook will stop. Hopefully this makes sense!
Is there a way to stop a runbook from running if parameters aren’t meant?
I wrote a post about scheduling a runbook to only run on the first business day of the month. You could use the same logic to specify (hard code) the months it's allowed to run using PowerShell. If it's not the correct month, the logic isn't met and the subsequent link would not let the runbook proceed.
Let me know if that helps. As far as stopping a runbook if parameters aren't met....do you literally mean if parameters in the initialize activity are not what is expected? For that you would also need PowerShell or the compare data activity to validate the parameters.
My company is tranfering many important files with Orchestrator 2012 R2 and we are using many different systems such as Windows, Linux, Solaris... For Linux/Solaris file transfers we are using the v1.3 FTP IP that you've made, but we have a problem. The List Folder is showing everything in a folder, (command ls -l) and we just need it to show only filenames, no dates or permissions, is there any workaround for this?
The List Folder activity is pretty limited as it is just doing a "dir $Path". I just tested trying to include a switch prefixing the path in the activity, but it gets interpreted as part of the path. If you're strictly connecting to internal systems, you could alternatively use the Run SSH Command to replace the List Folder activity. In that you could specify to return only files.
Do you have a github repo that you can put it in and share the repo link?
I've also changed my method for querying running/pending jobs quite a bit since that post :)
Thanks! Jon
$url = "https://webservice.domain.net:8443/Orchestrator2012/Orchestrator.svc/Jobs()?`$filter=(Status eq 'Running' or Status eq 'Pending')&`$select=Id,RunbookId,LastModifiedTime,Status"
Does the runbook actually stop? I thought this was a bug in the web service (which was later fixed in a rollup). The cancel/stop call did throw an error, but the runbook *does* actually stop.
Hi Jon,
ReplyDeleteI wanted to shoot you a quick note because I'm hoping you can help me. I work for a company called Windward IT Solutions and we specialize in implementing SCORCH along with a few other workflow development tools. Right now we are in desperate need of another automation expert. I came across your blog and it evident that you have a lot of expertise with SCORCH and runbook automation.
I was hoping you might be able to point me in the direction of a colleague that might be interested in the position and learning some of the other automation tools out there.
I'd appreciate any help or suggestions you can provide as to how I can get in touch with some people that have experience with SCORCH. If this is something you're interested in as well I'd be happy to speak with you about the opportunity.
Thanks,
Chris Werner
Technical Recruiter
Windward IT Solutions
703 812 0117
Jon,
DeleteForgot to include my email address: cwerner@windwardits.com
Jon,
ReplyDeleteMany thanks for the FTP Integration pack. Worked like a charm straight out of the box.
Using it for pulling system logs over SSH.
Thanks again
Ben
We are using SCORCH 2012 R2. We still see some orphaned jobs on the Monitor runbooks.
ReplyDeleteIs there any ways you know to fix it?
We have already implemented the steps in this link
http://www.concurrency.com/blog/how-to-clear-orphaned-orchestrator-runbook-instances/
Hi,
DeleteCan you confirm that you've actually stopped the running monitor runbooks with the orphaned jobs? The stored proc will only removed the orphaned instances if the runbook is NOT running. I don't think I've encountered a situation where the runbook has been stopped and the stored proc doesn't remove them.
I've used these queries back in the Opalis days to clear up the orphaned instances (update statements being commented out).
****Find and remove orphaned policies****
select *
from dbo.policyinstances
where timeended is null and status is null
--update dbo.policyinstances
--set [timeended] = getdate()
--set [status] = 'warning'
--where timeended is null and status is null
****Find and remove orphaned runbooks for specific runbook****
select POLICYINSTANCES.UniqueID, POLICIES.Name
from dbo.policyinstances, dbo.POLICIES
where status is null and (POLICYINSTANCES.PolicyID = POLICIES.UniqueID) and POLICIES.Name = ''
Order By PolicyID
--update policyinstances
--set policyinstances.[timeended] = getdate()
-- ,policyinstances.[status] = 'warning'
--From POLICYINSTANCES, POLICIES
--where policyinstances.status is null and (POLICYINSTANCES.PolicyID = POLICIES.UniqueID) and POLICIES.Name = ''
On a side note....You don't necessarily have to go into SQL to run the stored proc to clear orphaned instances. Right-clicking the management server in the Designer console and selecting "Log Purge" and Purge Now will also run that stored proc to remove the orphaned instances.
Hope that helps....
Jon
Forgot to mention....before running each of the queries above.
Delete1st Query - ALL runbooks need to be stopped prior to running.
2nd Query - The particular runbook to clean up orphaned instances needs to be stopped.
Yes we had to stop runbooks and clear orphans.
ReplyDeleteBut is there a way to get permanent fix for this, does microsoft have a solution to avoid orphans?
It is creasing our runbook servers sometimes, our runbooks stop working completely. Any ideas to fix it?
Hi,
DeleteOrphaned jobs are created (most of the time....) when runbook servers have connectivity issues with the database. The runbook server will go into a sleep mode for three minutes wherein all running runbooks will cease to run on that particular server and start running on another runbook server. After three minutes, the runbook server will continue to accept requests to run jobs.
Orphaned instances don't really cause any performance issues (to my knowledge) other than being annoying while still visible.
I don't foresee Microsoft changing any code to prevent the orphaned instances altogether....especially when the stored procedure is in the database to clear orphaned instances once the runbook is stopped.
Jon
Hey Im back again. same person.. can you reply to the above comment if possible.
ReplyDeletehttp://jmattivi.blogspot.com/2012/06/high-availability-for-runbook-servers.html
you have mentioned here the global variables have high availability.
We have situation in SCORCH R2, where the jobs pile up.
We have 8 runbooks servers. Each runbook server can take 100 jobs max load.
When 3 runbook servers hit their max threshold, the performance of SCORCH becomes slow.
hear teams mailing us alerts are not processed. can you help in this case?
Hi,
DeleteIt's never a good thing when runbook servers are hitting their max limit.
In this case, the solution would be to either
Change the target runbook server (other than one of the three) for a bunch of the runbooks to spread the load out that they're not getting maxed out.
OR
Increase the max jobs for the runbook servers to something acceptable for your needs (above 100).
Jon
Hi Jon,
ReplyDeleteI was wondering if had any luck using the decrypt activity in orchestrator 2012? I configured it exactly how it is in opalis but it fails stating :
Error Summary: public keyring not found
Details:
Secret keyring not found.
regards,
Jojo
Hi Jojo,
DeleteDoes the passphrase contain any special characters? I ran into that issue a while ago that produced a similar error, I believe, was classified as a bug.
Let me know,
Jon
The passphrase does have a "*".
DeleteYou could try creating a test set of keys to decrypt and create a passphrase with uppper and lower case characters (I think numbers were fine as well) to test. But I would guess that the "*" is creating the problem.
DeleteJon
I really would rather not pursue that since it would involve contacting each vendor we receive files from and the work to do that would be a major pain. Do you know of any other custom decrypt integration packs that I can try doing this with?
DeleteTechnically, you would need to generate new keys and just provide the vendor the new public key to use going forward. The new passphrase would then be updated in the runbooks.
DeleteOtherwise, I believe the only option would be to use the Run Program activity with gpg from the command line.
Jon
I figured out how to get this to work. I isolated our secure and public ring files in one directory and pointed the decrypt activity keyring folder setting to that directory and that did the trick. Apparently opalis is smart enough to pair the appropriate files together to decrypt files but orchestrator isn't.
DeleteRegards,
Jojo
Awesome! Thanks for posting back! I can't imagine there were any changes to the PGP activities between Opalis and Orchestrator....but at least there's a workaround.
DeleteThanks again for posting back!!
Jon
Hi Jon,
ReplyDeleteI have 4 runbooks designed to handle quarterly audits, however I would like them to kick off automatically. I've tried using the monitor/check schedule, but to only start quarterly means I’m manually adding in exceptions (I’m trying to manual maintenance). One idea I had was to have my check schedule set on one day per month, then add an if statement in my first .Net script object (using Powershell). So if the current month variable doesn’t match the audit months, the whole runbook will stop. Hopefully this makes sense!
Is there a way to stop a runbook from running if parameters aren’t meant?
Hello,
DeleteI wrote a post about scheduling a runbook to only run on the first business day of the month. You could use the same logic to specify (hard code) the months it's allowed to run using PowerShell. If it's not the correct month, the logic isn't met and the subsequent link would not let the runbook proceed.
http://jmattivi.blogspot.com/2013/04/schedule-workaround-first-business-day.html
Let me know if that helps. As far as stopping a runbook if parameters aren't met....do you literally mean if parameters in the initialize activity are not what is expected? For that you would also need PowerShell or the compare data activity to validate the parameters.
Jon
Hi Jon,
ReplyDeleteMy company is tranfering many important files with Orchestrator 2012 R2 and we are using many different systems such as Windows, Linux, Solaris... For Linux/Solaris file transfers we are using the v1.3 FTP IP that you've made, but we have a problem. The List Folder is showing everything in a folder, (command ls -l) and we just need it to show only filenames, no dates or permissions, is there any workaround for this?
Kind regards,
Leon
Hi Leon,
DeleteThe List Folder activity is pretty limited as it is just doing a "dir $Path". I just tested trying to include a switch prefixing the path in the activity, but it gets interpreted as part of the path. If you're strictly connecting to internal systems, you could alternatively use the Run SSH Command to replace the List Folder activity. In that you could specify to return only files.
Jon
Thanks for your fast reply! I will try using the Run SSH Command.
ReplyDeleteLeon
I have an update for your script to get jobs that are running. Would like to send it to you. How can I do that?
ReplyDeleteHi Mike,
DeleteDo you have a github repo that you can put it in and share the repo link?
I've also changed my method for querying running/pending jobs quite a bit since that post :)
Thanks!
Jon
$url = "https://webservice.domain.net:8443/Orchestrator2012/Orchestrator.svc/Jobs()?`$filter=(Status eq 'Running' or Status eq 'Pending')&`$select=Id,RunbookId,LastModifiedTime,Status"
$results = Invoke-RestMethod -Method Get -Uri $url -Credential $creds -ContentType "application/atom+xml" -UseBasicParsing
$runbooks = @()
foreach ($result in $results)
{
Clear-Variable -Name runbook -ea SilentlyContinue
$runbook = New-Object psobject
$runbook | Add-Member -MemberType NoteProperty -Name Id -Value $($result.content.properties.Id."#text")
$runbook | Add-Member -MemberType NoteProperty -Name RunbookId -Value $($result.content.properties.RunbookId."#text")
$runbook | Add-Member -MemberType NoteProperty -Name LastModifiedTime -Value $($result.content.properties.LastModifiedTime."#text")
$runbook | Add-Member -MemberType NoteProperty -Name Status -Value $($result.content.properties.Status)
$runbooks += $runbook
}
Hi Mike,
DeleteDoes the runbook actually stop? I thought this was a bug in the web service (which was later fixed in a rollup). The cancel/stop call did throw an error, but the runbook *does* actually stop.
Jon