[ale] Any ftps / lftp experts?

[Lightner, Jeffrey]

OK I believe I have this sorted.   Thanks to those that responded.

It appears we have to do ftps explicit vs implicit.    Testing via WinSCP helped confirm that.

The script for running this in cron still needs to be finalized but I’m sharing what I have at this point for lftp to get ftps.   This is on RHEL6 by the way.

Some notes for the archives:
For lftp below:  -d = debug, -v = verbose, -c = command

# List specific file - Single command line
lftp -dv -c 'open -e set ftp:ssl-force true; set ftp:ssl-protect-data true; set ssl:verify-certificate false;open ftps://<USER>:<PASSWORD>@<URL>; ls <subdir>/<filename>'
In the above one can add multiple set commands separated by semicolon.

# List all files in a directory - Single command line
lftp -dv -c 'open -e set ftp:ssl-force true; set ftp:ssl-protect-data true; set ssl:verify-certificate false;open ftps://<USER>:<PASSWORD>@<URL>; ls <subdir>/*'

# List all files in a directory – using multiple rather than single lines (start/end with double quote) for scripting.  Specify port to use (21 = default for explicit, 990 = default for implicit, some partners specify alternate ports for these).  Note space used between user and password rather than colon.  Also URL here does NOT start with “ftps://” just the site name.   Use the name rather than IP because the SSL is issued to name (or set the option to ignore SSL errors.)
lftp -dv -c "
set ftp:ssl-force true
set ftp:ssl-protect-data true
set ssl:verify-certificate false
open <URL>:21
user <USER>  <PASSWORD>
ls <subdir>/*
bye
“

# List all files in a directory - Multiple lines with variables
## Set the variables
REMHOST=<URL>
REMPORT=21
REMUSER=<USER>
REMPASS=<PASSWORD>
REMDIR=<subdir>
## Use the variables
lftp -dv -c "
set ftp:ssl-force true
set ftp:ssl-protect-data true
set ssl:verify-certificate false
open $REMHOST:$REMPORT
user $REMUSER $REMPASS
ls $REMDIR/ *
bye
"


# List all files in a directory using variables.  Also get one of the files.  (Note mget works for multiple files instead of get used for single file.)
# Set the variables
# List all files in a directory - Multiple lines with variables
## Set the variables
REMHOST=<URL>
REMPORT=21
REMUSER=<USER>
REMPASS=<PASSWORD>
REMDIR=<subdir>
## Use the variables
lftp -dv -c "
set ftp:ssl-force true
set ftp:ssl-protect-data true
set ssl:verify-certificate false
open $REMHOST:$REMPORT
user $REMUSER $REMPASS
ls $REMDIR/ *
get $REMDIR/<filename>
bye
"

From: Boris Borisov <bugyatl at gmail.com>
Sent: Monday, December 03, 2018 3:08 PM
To: Lightner, Jeffrey <JLightner at dsservices.com>; Atlanta Linux Enthusiasts <ale at ale.org>
Subject: Re: [ale] Any ftps / lftp experts?

Not expert on ftp but what about for passive or ftp active mode.

I service a vendor for whom ftp active mode would not work. Ftp client has to be set explicitly in passive mode.

Maybe I'm shooting in the dark in your case.

On Mon, Dec 3, 2018, 2:56 PM Lightner, Jeffrey via Ale <ale at ale.org<mailto:ale at ale.org> wrote:
We have a new partner insisting they can only accept ftps (i.e. no sftp) logins for us to download files from their site.

I had found some time ago that the tool lftp can be used to login to ftps sites and had worked out a syntax that worked for one vendor.

On trying that syntax on this site it doesn’t work.   I found an alternate syntax on RedHat’s site and that worked for me (once) on 16-Nov to actually download files.

However, ever since then every time I try the newer syntax it fails.   Worse yet they lock the vendor is locking the account after 3 failed password attempts (which they says is the only reason) so every time I try this syntax the account is getting locked and I have to wait for them to unlock it.   Prior to running the syntax just doing a simple login works but of course doesn’t afterwards.  It appears the way lftp works is that even though one gives it the user/login at start it is actually logging in separately for each command it runs.

The basic command syntax here logs in successfully (until the account gets locked):
lftp -d -u <USER>,<PASSWORD> <ftpssite>

That puts me at an lftp prompt.   At that prompt I can send commands to change directory, list files, and even change the password.  However, if I try a “get <file>” it just times out and retries over and over.

The syntax (one liner) that worked for me on 16-Nov that now fails:
lftp -v -c 'open -e "set ftps:initial-prot """; set ftp:ssl-force true; set ftp:ssl-protect-data true; set ssl:verify-certificate false; open ftps://<USER>:<PASSWORD>@<ftpssite>; ls; cd outbound/<subdir>; ls; get <filename>'

I had later found things such as specifying port and also that the “set ftp:inititial-prot” was unsupported (I’d seen an error in output using it even though it had been successful downloading.)  I’d added the port option and removed the unsupported option that same day and they had worked but later in day the account was locked so further testing failed.   After having partner unlock I did additional testing but eventually it gets locked so I’ve never yet been able to download a file despite initial connections being successful.

Every time partner unlocks the account the basic login works:
lftp -d -u <USER>,<PASSWORD> <ftpssite>
But every time I try to use the expanded syntax the account gets locked again.   I can see it is showing failed login multiple times in the output but don’t know why it is doing that now when it worked previously.

I need to be able to script this so that we can login to the server with the user and password to see what files, if any, are available to download, then download the ones found.   This kind of operation is something I’ve done in the past with regular here docs for regular ftp and with perl with ftp as well as scripts for sftp.

Does anyone have syntax that works for similar purposes that would be willing to share?  I’m not married to the idea of lftp – that is just what I find suggested in most posts.   Is there a way to do it with wget?  curl? Some other CLI tool?

Remote only supports ftps – please don’t suggest sftp as that is simply not an option.

CONFIDENTIALITY NOTICE: This e-mail may contain privileged or confidential information and is for the sole use of the intended recipient(s). If you are not the intended recipient, any disclosure, copying, distribution, or use of the contents of this information is prohibited and may be unlawful. If you have received this electronic transmission in error, please reply immediately to the sender that you have received the message in error, and delete it. Thank you

_______________________________________________
Ale mailing list
Ale at ale.org<mailto:Ale at ale.org>
https://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.ale.org/pipermail/ale/attachments/20181211/676cb20e/attachment.html>