I've just released v1.3 of the FTP/SFTP IP.
Post:
http://jmattivi.blogspot.com/2012/09/scorch-ftp-integration-pack-update-v13.html
CodePlex:
https://scorchestrator.codeplex.com/releases/view/94807
********Please see this post for the updated v1.2
http://jmattivi.blogspot.com/2012/06/ftp-integration-pack-update-v11.html
********
I've just published an FTP/SFTP Integration Pack that can be downloaded from https://scorchestrator.codeplex.com/releases/view/89427.
Here are the current activities supported in the IP. The SFTP portion is based off of Putty's psftp software (documentation can be referenced here). The FTP functionality is based off of the System.Net.FtpWebRequest Class (documentation can be referenced here). I was looking to provide roughly the same functionality the FTP Integration Pack for Opalis had.
All fields are required for each activity. Secure and AutoAcceptKey are set to True by default
Secure = True
--SFTP
Secure = False
--FTP
AutoAcceptKey only applies if Secure = True
AutoAcceptKey = True
--Automatically accept the host key presented by the server
****This can be insecure****
AutoAcceptKey = False
--This will technically cause the application to prompt to accept the key, which in turn would cause the activity to fail. However, you can proactively manually store the host key and leave this set to false if security requires it.
Unfortunately, the FTPWebRequest class doesn't support wildcards for Secure = False (FTP). However, the Secure = True method (SFTP) in the IP does support using wildcards such as * or *.txt.
One way to get around the FTP method using wildcards would be to use the List Folder activity to find the files in the folder, parse them, and then pass each into the Download File activity.
For instance, use List Folder to get the files in the Output published data:
07-12-12 08:56AM 0 asdf.txt
07-12-12 08:56AM 0 blah.txt
Parse the Output and pass asdf.txt and blah.txt to the Download File activity.
Upload File Example:
Download File Example:
Please kick the tires on this and let me know if you find any bugs or see the potential for functionality improvements.
This is the first time I've really poked around w/ the OIT command line activity wizard....now I have to teach myself C# here shortly to start building projects in Visual Studio w/ the SDK going forward. :)
Hi,
ReplyDeleteThis is a great IP which was greatly needed!
I'm using an SFTP setup with a public/private key, so the destination server has the public key and action server will have the private key. Using this setup I can log into to SFTP server with the username and private key (no password). Does your IP support this method of connection, and if so where would I need to place the private key?
Thanks.
Hi DA,
DeleteCurrently the IP does not support using a private key in lieu of a password. The System.Net.FtpWebRequest class does not support this. The secure version of the IP relies on psftp.exe. I believe to use this, you need to use a saved Putty session and specify the ppk file as the private key.
Does that sound like it would apply to your use case? If so, I can look into updating the code to include the private key and disregard the password if specified.
Hi Jon,
ReplyDeleteNice work! I was just wondering if there was a simple way to use your IP to download all the files in a specific folder? Placing * in the FileName field doesn't seem to work.
Am I overlooking something?
Great thanks in advance!
Matthias
Hi Matthias,
DeleteI take it you're using the Secure = False (FTP) method to download the files?
Unfortunately, the FTPWebRequest class doesn't support wildcards. However, the Secure = True method in the IP does support using wildcards such as * or *.txt. I apologize....I did find this when I was building it and forgot to update the documentation in the blog POST (I'll do that now, so thanks for the reminder :)).
One way to get around the FTP method using wildcards would be to use the List Folder activity to find the files in the folder, parse them, and then pass each into the Download File activity.
For instance, use List Folder to get the files in the Output published data:
07-12-12 08:56AM 0 asdf.txt
07-12-12 08:56AM 0 blah.txt
Parse the Output and pass asdf.txt and blah.txt to the Download File activity.
Hi Jon,
ReplyDeleteIs there a way to use a different port? I tired host:port and -p port, but didn't work.
Hi,
DeleteIn the server name field, can you try....
name -P #
The port switch has to be an upper case "P". let me know if you're successful.
Hi Jon,
DeleteI hope you still monitor this forum.
I have started using this for SFTP transfers, but one of the remote servers I want to use has a different port. Whenever I use "hostname -P 221" the connection seems to remove all spaces and gives the error
"Looking up host "hostname-P221" ssh_init: Host does not exist"
Is there a workaround for this?
Thanks
Adam
Hi Adam,
DeleteCan you try setting the server parameter to this below? You're basically adding " -P 221" immediately after the server name (including the quotes). Please let me know if this workaround is successful. Thanks!
servername.domain.com" -P 221"
Hi Jon
ReplyDeleteHave you made any progress on DA's case? I have a similar scenario where the destination server holds my public key and I present my private key (*.ppk) on connection and also using a username and password. Currently use a script to call winscp and then a command file after connection is made.
Thanks
S.
Hi S,
DeleteUnfortunately, I haven't updated the IP...."yet" to include the use of key authentication. In cases like this, I've modified the source code to accomodate the psftp switches to allow for key auth and used it directly in the Run .Net Script activity.
You can use the CLA Wizard to load the assembly from the FTP IP to retrieve the source code. Please let me know if you have any issues or questions and I can give you a hand.
Jon
Once you have the source code, you'd create another variable in PowerShell for the location of your private key and then modify the psftp switches as below (careful if wrapped....it's all one line after $Output).
Delete#Run psftp.exe to Upload File - Auto Accept Host Key
$Output = $cmd | & "C:\Program Files (x86)\Common Files\Microsoft System Center 2012\Orchestrator\Extensions\Support\Integration Toolkit\90ac7e38-b93a-4da7-982b-912e06e2ebb2\psftp.exe" -v -pw $Passphrase $Username@$SourceServer -P 822 -i $PrivateKey 2>&1
I have a error and my password is correct
ReplyDeleteFile Failed to Transfer!!!!
Looking up host "53.31.101.22" Connecting to 53.31.101.22 port 22
Server version: SSH-2.0-1.82_sshlib GlobalSCAPE
Using SSH protocol version 2
We claim version: SSH-2.0-PuTTY_Release_0.62
Doing Diffie-Hellman group exchange
Doing Diffie-Hellman key exchange with hash SHA-1
Host key fingerprint is:
ssh-rsa 2048 51:63:82:06:be:0d:49:74:69:19:90:3f:69:fe:78:1f
Initialised AES-256 CBC client->server encryption
Initialised HMAC-SHA1 client->server MAC algorithm
Initialised AES-256 CBC server->client encryption
Initialised HMAC-SHA1 server->client MAC algorithm
Using username "mpc_es_prod
".
Attempting keyboard-interactive authentication
Keyboard-interactive authentication failed
Access denied
Disconnected: No supported authentication methods available (server sent: )
Fatal: Disconnected: No supported authentication methods available (server sent: )
Exception: RuntimeException
Target site: PipelineBase.Invoke
Stack trace:
at System.Management.Automation.FlowControlNode.Execute(Array input, Pipe outputPipe, ExecutionContext context)
at System.Management.Automation.ParseTreeNode.Execute(Array input, Pipe outputPipe, ArrayList& resultList, ExecutionContext context)
at System.Management.Automation.StatementListNode.ExecuteStatement(ParseTreeNode statement, Array input, Pipe outputPipe, ArrayList& resultList, ExecutionContext context)
Hello,
ReplyDeleteI am trying to use this IP because the one in orchestrator is not working properly. I was directed to try you IP by microsoft support. But it isn't working when I attempt to upload, I received the below "error summary text". Any help you can provide is much appreciated.
regards,
Jojo
File Failed to Transfer!!!!
Looking up host "ftp.waproviderone.org
"
ssh_init: Host does not exist
Exception: RuntimeException
Target site: PipelineBase.Invoke
Stack trace:
at System.Management.Automation.FlowControlNode.Execute(Array input, Pipe outputPipe, ExecutionContext context)
at System.Management.Automation.ParseTreeNode.Execute(Array input, Pipe outputPipe, ArrayList& resultList, ExecutionContext context)
at System.Management.Automation.StatementListNode.ExecuteStatement(ParseTreeNode statement, Array input, Pipe outputPipe, ArrayList& resultList, ExecutionContext context)
Hi Jojo,
DeleteCan you confirm that there is not a space or return at the end in the "DestServer" parameter? I only ask since the end quote above that you posted is on a new line.
Does the server limit connections by IP address? If so and they haven't explicitly added your connecting IP, it would be blocked from connecting to the destination server.
Let me know,
Jon
Nice! That was it. Thanks for you help.
Deleteregards,
Jojo
Great!....Glad you got it working!! :)
DeleteJon, I have a question similar to what was asked above. I'm wondering if it's an issue with the SFTP server using FreeFTP?
ReplyDeleteFile Failed to Transfer!!!!
Looking up host "host" Connecting to "host" port 22
Server version: SSH-2.0-WeOnlyDo-wodFTPD 3.2.2.395
We believe remote version has SSH-2 rekey bug
Using SSH protocol version 2
We claim version: SSH-2.0-PuTTY_Release_0.62
Using Diffie-Hellman with standard group "group14"
Doing Diffie-Hellman key exchange with hash SHA-1
Host key fingerprint is:
"xxxx"
Initialised AES-256 SDCTR client->server encryption
Initialised HMAC-SHA1 client->server MAC algorithm
Initialised AES-256 SDCTR server->client encryption
Initialised HMAC-SHA1 server->client MAC algorithm
Using username "username
".
Further authentication required
Further authentication required
Sent password
Password authentication failed
Amber
Hi Amber,
DeleteIs the destination user setup for private key authentication rather than password or something like that? Are you able to use FileZilla or psftp outside of Orchestrator to test the authentication?
Let me know.
Jon
Jon, did you ever get to updating the IP to include private key authentication? If not, do you now of any IPs that do so?
ReplyDeleteHi Andrew,
DeleteUnfortunately, I have not updated the IP to include the key auth yet....
Here is the code though that you can use which integrates with the Integration Pack. You should be able to plug this directly into the Run .Net Script activity and provide the parameters to use for the key auth. Let me know if you have any issues. Hope this helps.
Jon
#DownloadFile#################################
#Created By: Jon Mattivi
#Modified Date: 09/18/2012
#v1.3 SCOrch IP for SFTP/FTP
############################################
#Define Variables
$SourcePath = ''
$SourceServer = ''
$DestPath = ''
$FileName = ''
$Username = ''
$Passphrase = ''
$PrivateKey = ''
$Secure = $true
$AutoAcceptKey = $true
$UsePassive = $true
$UseBinary = $true
$Output = @()
If ($Secure -eq $true)
{
If ($AutoAcceptKey -eq $true)
{
$cmd = @(
"y",
"This is a really, really, really, really long bogus cmd",
"lcd ""$DestPath""",
"cd ""$SourcePath""",
"mget ""$FileName""",
"bye"
)
#Run psftp.exe to Upload File - Auto Accept Host Key
$Output = $cmd | & "C:\Program Files (x86)\Common Files\Microsoft System Center 2012\Orchestrator\Extensions\Support\Integration Toolkit\90ac7e38-b93a-4da7-982b-912e06e2ebb2\psftp.exe" -v -pw $Passphrase $Username@$SourceServer -P 822 -i $PrivateKey 2>&1
$Err = [String]($Output -like "*=>*")
$LocalErr = [String]($Output -like "*New local directory is*")
$RemoteErr = [String]($Output -like "*Remote directory is now*")
If ($LastExitCode -ne 0)
{
throw "File Failed to Transfer!!!! `n $($Output)"
}
ElseIf (($LocalErr.Contains("New local directory is")) -eq $false)
{
throw "Failed to Change Local Directory!!!! `n $($Output)"
}
ElseIf (($RemoteErr.Contains("Remote directory is now")) -eq $false)
{
throw "Failed to Change Remote Directory!!!! `n $($Output)"
}
ElseIf (($Err.Contains("=>")) -eq $false)
{
throw "File Failed to Transfer!!!! `n $($Output)"
}
}
}
#Create Published Data
$Output = [system.string]::Join("`n", $Output)
$MyObject = new-object psobject
$Props = @"
SourceServer
SourcePath
DestPath
FileName
Username
Secure
AutoAcceptKey
Output
"@
$Props -split "`n" | %{
$MyObject | add-member -membertype noteproperty -name $_.trim() -value $null
}
$MyObject.SourceServer = $SourceServer
$MyObject.SourcePath = $SourcePath
$MyObject.DestPath = $DestPath
$MyObject.FileName = $FileName
$MyObject.Username = $Username
$MyObject.Secure = $Secure
$MyObject.AutoAcceptKey = $AutoAcceptKey
$MyObject.Output = $Output
$MyObject
Hi,
ReplyDeleteI'm having some problems with the Upload File activity in the SFTP Integration Pack.
I want to go into a folder but without having to use "/". For example:
cd "download", it won't do this for some reason it always adds the "/", how can I bypass that?
Hi Leon,
DeleteThe code is adding the "/" so that if it were to be omitted that the directory change is still successful. Is it failing to change the destination upload path to "download"? Is the full path for the upload "/download/"?
Jon
Perfect, thank you very much.
ReplyDeleteAdam
Hi Jon, I am trying to use port 21 and it doesn't matter how I input it. I tried the hostname" -P 21" as mentioned above and also tried the IP:21 and that didn't work either. It keeps assuming port 22. Help! Contact me at jdoty@flowcorp.com
ReplyDeleteHi Jeff,
DeleteCan you confirm if you're trying to use regular FTP, not an SFTP (secure) transfer? If that's the case, you would specify the parameter "Secure" to "False". That would default the transfer to port 21 using FTP.
Let me know if that helps.
Jon
Hi Jon,
Deleteproblem is that it is secure over 21...I didn't set this up and I am not sure why this was setup this way. I am connecting to a LiquidFiles server.
Oops, I think it worked setting "Secure" to "False". I need to check that the client got the file.
ReplyDeleteHi Jon,
ReplyDeleteBack again, I am getting this error from the FTP Upload File IP activity:
Exception calling "ReadAllBytes" with "1" argument(s): "Could not find file '\\Shapetechnologies.com\Root\Flow\Operations\Seattle\Quality\TXT RESULTS\TXT RESULTS'."
Exception: MethodInvocationException
Target site: PipelineBase.Invoke
Stack trace:
at System.Management.Automation.ExceptionHandlingOps.ConvertToMethodInvocationException(Exception exception, Type typeToThrow, String methodName, Int32 numArgs, MemberInfo memberInfo)
at CallSite.Target(Closure , CallSite , Type , Object )
at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
at System.Management.Automation.Interpreter.DynamicInstruction`3.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
Even though it says it can't find the file, I have even reset everything to look in the C:\temp folder for the file and it gives me the same error. So I am resorting to asking you. Sorry.
Jeff
Hi Jeff,
DeleteCan you confirm if you're using ftp or sftp? Is the source path above a unc path accessible over 445? Also, can you confirm that the service account the IP is running as has access to that path?
Hi Jon,
DeleteUsing FTP, I have verified that the account is working by connecting with FileZilla from my automation (Orchestrator) server (FTP). I can connect to the LiquidFiles FTP server via web browser which runs Https as well. The weird thing is that this was all working last Friday before the move. The SCORCH server and the LiquidFiles FTP server are in the same site, one in a DMZ and the other on the regular network. Same FW rules between them.
Hi Jon,
DeleteSome more info. Here is the connection log for FileZilla:
Status: Resolving address of fta.shapetechnologies.com
Status: Connecting to 65.122.178.150:21...
Status: Connection established, waiting for welcome message...
Status: Initializing TLS...
Status: Verifying certificate...
Status: TLS connection established.
Status: Logged in
Status: Retrieving directory listing...
Status: Server sent passive reply with unroutable address. Using server address instead.
Status: Directory listing of "/" successful
Hi Jeff,
DeleteAre you able to use Putty's psftp to manually check the connection/transfer? The integration pack is using psftp, so manually running it should hopefully repro any issues.
You mentioned the runbook was working "before the move". What move or what change was made which this stopped working after?
Hi Jon,
ReplyDeleteI can load Putty and try the connection, I posted my FileZilla connection log above (which works). Here is my Orchestrator Options FTP settings:
FTP connection settins in Orch -
Connection Type: NormalWithExplicitTlsSsl31Authentication
Transfer Type (FTP): Passive
Server: fta.shapetechnologies.com
Port: 21
Username:Yes, used the same with FileZilla
Password:Yes, used the same with FileZilla
Upload File Properties -
SourcePath: C:\Temp\Test\
DestServer: fta.shapetechnologies.com
DestPath:
Filename: File.txt
Username: username
Password: password
Secure: False
AutoAcceptKey: True
UsePassive: True
UseBinary: True
The Move I am taking about is that we moved our inhouse servers/san/etc to a co-location facility.
I tried the psftp connection and it doesn't do anything and then closes the window. I tried the following commands:
open fta.shapetechnologies.com -l:username -p:21 -pw:password
open: invalid port number
open fta.shapetechnologies.com -l:username -pw:password
open: invalid port number
Sorry for the late response. I forgot that you said this was "ftp" not "sftp" so psftp wouldn't be involved.
DeleteI just noticed that the source path is a local path "C:\Temp\Test\", can you confirm that path and source file is on the runbook server and not remote?
Can you try updating the variables in this code and run it for the ftp upload?
$SourcePath = 'C:\Temp\Test\'
$DestServer = 'fta.shapetechnologies.com'
$DestPath = ''
$FileName = 'File.txt'
$Username = ''
$Password = ''
$Secure = $false
$AutoAcceptKey = $true
$UsePassive = $true
$UseBinary = $true
$Output = @()
$uri = "ftp://$DestServer/$DestPath/$FileName"
#Create FTP Rquest Object
$FTPRequest = [System.Net.FtpWebRequest]::Create($uri)
$FTPRequest = [System.Net.FtpWebRequest]$FTPRequest
$FTPRequest.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$FTPRequest.UseBinary = $UseBinary
$FTPRequest.UsePassive = $UsePassive
$FTPRequest.Timeout = 120000
$FTPRequest.Credentials = new-object System.Net.NetworkCredential($Username, $Password)
#Read the File for Upload
$FileContent = [System.IO.File]::ReadAllBytes($SourcePath + "\" + $FileName)
$FTPRequest.ContentLength = $FileContent.Length
#Get Stream Request by bytes
If ($FileContent){
$Run = $FTPRequest.GetRequestStream()
$Run.Write($FileContent, 0, $FileContent.Length)
#Cleanup
$Run.Close()
$Run.Dispose()
}
Hi Jon,
ReplyDeleteIt looks like the problem was outside of the config for the FTP IP (Network). This makes more sense since it was working before the move and not after the move. So I spent a week chasing my tail. So I am sorry for troubling you, but it is nice to know that you are so responsive to questions. That means a ton to me, so thanks for your help.
Hi Jon,
ReplyDeleteI tried to download the IP for a new server and all I am getting is a bunch of .json files and other files that don't even have an extension. No MSI or even .oip files. So I am at a loss. Have been using your IP on an old Orch server (it was already installed). Love it by the way, now need it so I can install on the new server. Jeff Doty (jdoty@flowcorp.com)
Hi Jeff,
DeleteDid you get my previous reply on getting it from GitHub? Let me know if that works.
It is also available out on my GitHub repo (oip file and all source code). Let me know if you have any issues.
https://github.com/jmattivi/Orchestrator/tree/master/Integration%20Packs/FTP-SFTP
Hi Jon,
DeleteThanks, that worked. Sorry I somehow missed your previous response. Good to go now.
Hello, first of all thank you for your effort in developing this IP. We have some issue because of the old version of putty, I mean probably some new authentication protocol or so is not supported, this is an example:
ReplyDeleteFile Failed to Transfer!!!!
Looking up host "sftp.domain.com" Connecting to XXX.XXX.XX.XXX port 22
We claim version: SSH-2.0-PuTTY_Release_0.62
Server version: SSH-2.0-WS_FTP-SSH_8.6.0.1027
Using SSH protocol version 2
Doing Diffie-Hellman group exchange
Server unexpectedly closed network connection
Fatal: Server unexpectedly closed network connection
Exception: RuntimeException
Target site: PipelineBase.Invoke
Stack trace:
at System.Management.Automation.FlowControlNode.Execute(Array input, Pipe outputPipe, ExecutionContext context)
at System.Management.Automation.ParseTreeNode.Execute(Array input, Pipe outputPipe, ArrayList& resultList, ExecutionContext context)
at System.Management.Automation.StatementListNode.ExecuteStatement(ParseTreeNode statement, Array input, Pipe outputPipe, ArrayList& resultList, ExecutionContext context)
so we tryed overwriting putty to a newer version but did not work. Can you help us?
Thank you in advance
Hi,
DeleteCan you repro the issue or test a transfer outside of SCOrch/IP using psftp manually from the command line or PowerShell?
Hello, sorry for late answer,
Deletewe were not able to reproduce the very same error, maybe we are not using the very same commends that IP is passing to sftp
but it we try to simply connect to sftp wih version 0.62 we got this error:
psftp.exe username@domain.com
Fatal: Server unexpectedly closed network connection
and if we do the same with psftp version 0.74 we got this
psftp.exe username@domain.com
Using username "username"
username@domain.com's password:
so the new psftp seems to work fine with this psftp site
we already tryied to overwrite psftp.exe in the IP folder but this causes the List command to fail
Thank you for support
Hi Jon, would you consider releasing your "Integration Pack for FTP" to be able to deploy/run on Orchestrator 2022 as 64-bit?
ReplyDelete