Showing posts with label Integration Pack. Show all posts
Showing posts with label Integration Pack. Show all posts

Friday, June 14, 2013

Flatten Multi-Valued Published Data - Part 1

This will probably be a two-part post.

I recently made some enhancements to one of my scripts to take flattened published data and format it so that it's readable (while still flattened).

A little background where this might apply....

If you have many data values that need to be emailed out, it's not likely that you want to send an email for each value.  Ideally, all of the values (filenames, database fields, etc.) would be emailed in a readable format in one email.

I most often use this PowerShell script immediately after a Query Database activity that has published data flattened seperated by "__%__" (without quotes).  The Query Database activity would read multiple rows from a database that meet specified criteria.  This particular example uses the Standard Logging IP by Charles Joy.

The published data passed into the Run .Net Script is the "Full line as string with fields separated by ';'".

Ex. Table Query Results:









Ex. Full line as string.... flattened with __%__ Published Data (not very pretty!)

6/14/2013 3:46:02 PM;3 - Failure;Server4;ActivityName:  Copy File From A to B;ErrorSummary:  Access Denied__%__6/14/2013 3:44:25 PM;3 - Failure;Server3;ActivityName:  Copy File From A to B;ErrorSummary:  File Does Not Exist__%__6/14/2013 3:42:07 PM;4 - Completed;Server2;Copy Completed;__%__6/14/2013 3:41:05 PM;4 - Completed;Server1;Copy Completed;

Posh Script to get the published data:































The Run .Net Script activity also has the published data flattened, but with line breaks.  The table variable would then be sent in the email.

Ex.
6/14/2013 3:46:02 PM
3 - Failure
Server4
ActivityName:  Copy File From A to B
ErrorSummary:  Access Denied

6/14/2013 3:44:25 PM
3 - Failure
Server3
ActivityName:  Copy File From A to B
ErrorSummary:  File Does Not Exist

6/14/2013 3:42:07 PM
4 - Completed
Server2
Copy Completed

6/14/2013 3:41:05 PM
4 - Completed
Server1
Copy Completed


#######################################################################
$table = @()

$RawFailures = @'
<Published Data seperated by __%__>
'@

$Regex = [regex] '/*__%__'
$Failures = $Regex.Split("$RawFailures")
ForEach($Data in $Failures)
    {
    If ($Data.Split(";")[0].Length -ne 0) {
    $Field1 = $Data.Split(";")[0]}
    Else {$Field1 = $null}
    If ($Data.Split(";")[1].Length -ne 0) {
    $Field2 = $Data.Split(";")[1]}
    Else {$Field2 = $null}
    If ($Data.Split(";")[2].Length -ne 0) {
    $Field3= $Data.Split(";")[2]}
    Else {$Field3 = $null}
    If ($Data.Split(";")[3].Length -ne 0) {
    $Field4= $Data.Split(";")[3]}
    Else {$Field4 = $null}
    If ($Data.Split(";")[4].Length -ne 0) {
    $Field5= $Data.Split(";")[4]}
    Else {$Field5 = $null}

    IF ($Data.Length -ne 0) {
    $table += $Field1, $Field2, $Field3, $Field4, $Field5, ""
    }
    }
   
$table = @($table | Where-Object {$_ -ne $null})
#######################################################################

Friday, April 26, 2013

Schedule Activity: First Business Day of the Month

Maybe this is more straightforward than I originally thought it was....sometimes I tend to over think and complicate things than they actually are.  So just in case someone else finds this helpful.... :)

I recently got a request for a workflow to run on the 1st calendar day of the month or first business day if the 1st fell on a weekend.  I first though that it wasn't possible to do with the native scheduling and PowerShell was needed.  After I wrote up a Posh script to do the heavy lifting, it dawned on me how to use the native schedule activity.  It's not as straight forward as using the script, but does work.

Orchestrator's built-in schedule activity supports two options:
1)  Specifying the calendar day(s) of the month to run

2)  Specifying the week day to run

Here is the "main" runbook for the schedule that will invoke the rest of the workflow.  It requires four different schedule activities to accomplish the logic.  Each of the following links (green) have the logic "confirms to schedule equal to true".













Here is the PowerShell script to accomplish the same schedule result.  The "$Success" variable would be in the Published Data to use in the link logic to continue or not if Success equals true.

$DayOfWeek = [DateTime]::Now.DayOfWeek
$DayOfMonth = [DateTime]::Now.Day

If (  (($dayOfMonth -eq 1) -and ($dayOfWeek -ge [DayOfWeek]::Monday) -and ($dayOfWeek -le [DayOfWeek]::Friday)) -or ((($dayOfMonth -eq 2) -or ($dayOfMonth -eq 3)) -and ($dayOfWeek -eq [DayOfWeek]::Monday))  )
{
    $Success = $true   
}
Else {
    $Success = $false
}

Thursday, February 14, 2013

Orchestrator and Excel Automation

I figured I'd throw this out here in case anyone is looking to automate Excel tasks through Orchestrator and PowerShell and running into issues.  I was tasked with modifying an Excel file (removing worksheets and saving as a csv file) in SCOrch as part of a workflow prior to processing the file.

I used the PowerShell script below in the Run .Net Script activity.

Script
#Remove Worksheets
$objExcel = new-object -Com Excel.Application
$xlFixedFormat = [Microsoft.Office.Interop.Excel.XlFileFormat]::xlWorkbookDefault
$objExcel.Visible = $false
$objExcel.displayalerts = $false
$objWorkbook = $objExcel.Workbooks.Open("\\PathtoMyFile\MyFile.xlsx")
$objWorksheets = $objWorkbook.Worksheets | Where {($_.Name -ne "Data")}
If ($objWorksheets) {
    ForEach ($objWorksheet in $objWorksheets) {
        $objWorksheet.Delete()
    }
}
$objWorkbook.Save()
$objExcel.Workbooks.Close()
$objExcel.quit()
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($objExcel)
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($objWorkbook)
Remove-Variable objExcel
Remove-Variable objWorkbook

Running this script manually on the runbook server worked fine and as expected.  However, running the script in SCOrch throws the following error (or similar error with opening the file).

Error
Exception calling "Open" with "1" argument(s): "Microsoft Excel cannot access the file '\\PathtoMyFile\MyFile.xlsx'. There are several possible reasons:

• The file name or path does not exist.
• The file is being used by another program.
• The workbook you are trying to save has the same name as a currently open workbook."


To solve this issue, create an empty folder called "Desktop" under the systemprofile folder as shown below:

C:\Windows\SysWOW64\config\systemprofile\Desktop
C:\Windows\System32\config\systemprofile\Desktop 

Wednesday, November 21, 2012

Run .NET Script: Catching PowerShell Output Into the Current Session

Here is a handy tip for catching PowerShell output into a variable as Published Data.

There are often times you need to execute a cmd within PowerShell that creates output in the cmd's output, but not PowerShell.

I came across a similar issue on the technet forums.
http://social.technet.microsoft.com/Forums/en-US/scogeneral/thread/a9511617-c174-4a5b-a531-3110471c9222

For example....

Running w/ PowerShell in the Run .NET Script activity cannot catch the output of the winrs cmd.  Even w/ adding the $Output = ... and adding the Output variable to published data.

$Output = winrs -r:server_name -u:server_name\administrator -p password net localgroup Administrators Domain\User /ADD

This will result in the Output variable in the published data being empty.

The trick is simply adding "2>&1" (without quotes) at the end of the cmd.  So the complete command would look like this.

 $Output = winrs -r:server_name -u:server_name\administrator -p password net localgroup Administrators Domain\User /ADD 2>&1

This will result in the $Output variable catching the result from winrs into the published data.  Also note, you may have to Flatten the data since the result may end up in multiple lines.




Here is a link that explains different methods for catching output in PowerShell.
http://mctexpert.blogspot.com/2010/11/what-does-2-mean-in-powershell.html

Tuesday, September 18, 2012

SCOrch FTP Integration Pack Update v1.3

I've just released v1.3 of the Orchestrator IP for FTP/SFTP.  You can find it on codeplex at this link:


Please install this in a test environment before deploying to production!  Also, please comment on any issues/bugs you may encounter.

********
Since adding the extra "UsePassive" and "UseBinary" input fields, please note the following steps required for existing installs upgraded to v1.3.  These fields must be commited to the database in the existing activity before they will work.

1)  Stop runbooks where the activities are located.
2)  Check out the runbook
3)  Open the activity and click "Finish" (no changes required)
4)  Check in the runbook
(Thanks to Jeff for the insight for commiting the added fields to existing activities!)
********

Activity Change Log:
 
Create Folder
Resolve issue where host key is not saved and cmd to change directory fails
Add option for passive/active connection
Allow Paths/Filenames with spaces
Adjust FTP Timeout
Dir Change Error Handling

Delete File
Resolve issue where host key is not saved and cmd to change directory fails
Add option for passive/active connection
Allow Paths/Filenames with spaces
Adjust FTP Timeout
Dir Change Error Handling

Delete Folder
Resolve issue where host key is not saved and cmd to change directory fails
Add option for passive/active connection
Allow Paths/Filenames with spaces
Adjust FTP Timeout
Dir Change Error Handling

Download File
Resolve issue where host key is not saved and cmd to change directory fails
Add option for deleting source file(s)
Add option for bin/ascii transfer
Add option for passive/active connection
Allow Paths/Filenames with spaces
Adjust FTP Timeout
Dir Change Error Handling

List Folder
Resolve issue where host key is not saved and cmd to change directory fails
Add option for passive/active connection
Allow Paths/Filenames with spaces
Adjust FTP Timeout
Dir Change Error Handling

Rename File
Resolve issue where host key is not saved and cmd to change directory fails
Add option for passive/active connection
Allow Paths/Filenames with spaces
Adjust FTP Timeout
Dir Change Error Handling

Upload File
Resolve issue where host key is not saved and cmd to change directory fails
Add option for deleting source file(s)
Add option for bin/ascii transfer
Add option for passive/active connection
Allow Paths/Filenames with spaces
Adjust FTP Timeout
Dir Change Error Handling
Empty File/0 byte Error Handling

Friday, August 24, 2012

Run Program Activity issue with Server 2008

****
Update 9/7/2012
The issue lies at the OS level w/ the UAC settings.  The changes reflect the differences from 2008 to 2008 R2....
i.e. 
Run all administrators in Admin Approval mode
Behavior of the elevation prompt for administrators in admin approval mode
****

****
Update 8/30/2012
This appears to only happen when using alternate credentials in the "Run As" field.  I've also repro'd this back in Opalis, so it's not something w/ the updated Run Program activity in SCOrch.  More troubleshooting to come....
****

Yesterday I came across an issue w/ the Run Program standard activity.  I have installed the RU2 hotfix release late in July to resolve the issue running the activity against Server 2003/2008/2008SP2.

The activity does run "successfully" in the console.  However, it does not seem to be running the specified command or program in an elevated UAC session.

Pure Output from running "ipconfig /flushdns" via Run Program on a 2008 server:

The requested operation requires elevation.


While expected behavior on a 2008 R2 server:
Windows IP Configuration

Successfully flushed the DNS Resolver Cache.



 I've also posted the question to the TechNet forums to see if this is a known issue since RU2.
http://social.technet.microsoft.com/Forums/en-US/scogeneral/thread/b03cb650-4abc-44a6-a097-9c1bf4d979a9


Tuesday, June 19, 2012

FTP Integration Pack Update v1.2

****Update 20120918****
I've just released v1.3 of the FTP/SFTP IP.
Post:
http://jmattivi.blogspot.com/2012/09/scorch-ftp-integration-pack-update-v13.html
CodePlex:
https://scorchestrator.codeplex.com/releases/view/94807


 ****Update****
I've updated the IP to v1.2 to fix an issue w/ the List Folder activity when using Secure = False.  Updated build has been posted to codeplex.
https://scorchestrator.codeplex.com/releases/view/89913


I've kicked the tires on the FTP Integration Pack a bit more and have updated the List Folder activity.  It will now only output the directory contents rather than displaying the entire verbose output (which is kinda redundant from the results published data).

I've also updated the download to Stable from Beta.  Please let me know if any bugs are encountered and/or if there are any feature requests.

Unfortunately, the FTPWebRequest class doesn't support wildcards for Secure = False (FTP). However, the Secure = True method (SFTP) in the IP does support using wildcards such as * or *.txt.

One way to get around the FTP method using wildcards would be to use the List Folder activity to find the files in the folder, parse them, and then pass each into the Download File activity.

For instance, use List Folder to get the files in the Output published data:
07-12-12 08:56AM 0 asdf.txt
07-12-12 08:56AM 0 blah.txt

Parse the Output and pass asdf.txt and blah.txt to the Download File activity.


The Integration Pack can be downloaded from this link below:
https://scorchestrator.codeplex.com/releases/view/89913



Tuesday, June 12, 2012

Integration Pack for FTP

****Update 20120918****
I've just released v1.3 of the FTP/SFTP IP.
Post:

http://jmattivi.blogspot.com/2012/09/scorch-ftp-integration-pack-update-v13.html
CodePlex:

https://scorchestrator.codeplex.com/releases/view/94807

********Please see this post for the updated v1.2
http://jmattivi.blogspot.com/2012/06/ftp-integration-pack-update-v11.html
********

I've just published an FTP/SFTP Integration Pack that can be downloaded from https://scorchestrator.codeplex.com/releases/view/89427.

Here are the current activities supported in the IP.  The SFTP portion is based off of Putty's psftp software (documentation can be referenced here).  The FTP functionality is based off of the System.Net.FtpWebRequest Class (documentation can be referenced here).  I was looking to provide roughly the same functionality the FTP Integration Pack for Opalis had.



All fields are required for each activity.   Secure and AutoAcceptKey are set to True by default

Secure = True
 --SFTP

 Secure = False
 --FTP

AutoAcceptKey only applies if Secure = True
AutoAcceptKey  = True
--Automatically accept the host key presented by the server
****This can be insecure****

AutoAcceptKey = False
--This will technically cause the application to prompt to accept the key, which in turn would cause the activity to fail.  However, you can proactively manually store the host key and leave this set to false if security requires it.


Unfortunately, the FTPWebRequest class doesn't support wildcards for Secure = False (FTP). However, the Secure = True method (SFTP) in the IP does support using wildcards such as * or *.txt.

One way to get around the FTP method using wildcards would be to use the List Folder activity to find the files in the folder, parse them, and then pass each into the Download File activity.

For instance, use List Folder to get the files in the Output published data:
07-12-12 08:56AM 0 asdf.txt
07-12-12 08:56AM 0 blah.txt

Parse the Output and pass asdf.txt and blah.txt to the Download File activity.
 

Upload File Example:




Download File Example:



Please kick the tires on this and let me know if you find any bugs or see the potential for functionality improvements.

This is the first time I've really poked around w/ the OIT command line activity wizard....now I have to teach myself C# here shortly to start building projects in Visual Studio w/ the SDK going forward. :)