<div dir="ltr"><div>I suggest gzip (or a mutually agreeable archive format) the file structure and sending one<br></div>file...<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Nov 22, 2013 at 7:50 AM, Lightner, Jeff <span dir="ltr"><<a href="mailto:JLightner@water.com" target="_blank">JLightner@water.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-US">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Long directory structures involved. In fact on our initial attempt we found that it didn’t download everything because the default behavior of wget is to only
go down 5 levels so we had restarted with 99 levels the max it would allow. I don’t think we had any that actually hit 99 levels but we probably ought to verify that.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The find was a straight forward find with no flags initially. Later find for –type f was done then another more complicated one done just to show directories.
Adding those together resulted in the same total as the initial find and wget summary.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">We tried NLIST (LIST not available) but it doesn’t do recursion at the remote site.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:ale-bounces@ale.org" target="_blank">ale-bounces@ale.org</a> [mailto:<a href="mailto:ale-bounces@ale.org" target="_blank">ale-bounces@ale.org</a>]
<b>On Behalf Of </b>David Tomaschik<br>
<b>Sent:</b> Thursday, November 21, 2013 8:43 PM<br>
<b>To:</b> Atlanta Linux Enthusiasts<br>
<b>Subject:</b> Re: [ale] 117000 files vs 240 missing - amazon<u></u><u></u></span></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Is it all in one directory, or was there directory structure transferred? What were the predicates to your find command? (Thinking their count might've included directories or something.)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Thu, Nov 21, 2013 at 1:59 PM, Lightner, Jeff <<a href="mailto:JLightner@water.com" target="_blank">JLightner@water.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">A vendor put a site on Amazon with some files we need. We don’t have sftp access to this Amazon site but do have ftp access.
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Accordingly we did a wget to download all the files using our ftp credentials. When all done we got over 117,000 files and saw no errors in the wget.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">The problem is vendor is telling our director there are 240 more files in their count than we downloaded. This is less than a 0.2% difference so I suspect it has something to
do with the way they count vs. the way we did. (We used find piped to wc –l.) Our count matches the summary wget output when it finished so we are sure we’re correctly counting what wget did but of course it’s possible wget actually missed something though
it seems unlikely to me.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">The question is does anyone know what might cause such a difference? Alternative does anyone know another way we could count the files on the Amazon site using our ftp credentials
other than going in and counting them one by one?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">We’re trying to find out how the vendor did their count but I was hoping someone already knows of some vagary on Amazon sites that would cause this kind of discrepancy.<u></u><u></u></p>
</div>
<p> <u></u><u></u></p>
<p> <u></u><u></u></p>
<p> <u></u><u></u></p>
<p> <u></u><u></u></p>
<p><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:fuchsia">Athena</span><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:fuchsia">®</span><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:fuchsia">,
Created for the Cause</span><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:fuchsia">™
</span><u></u><u></u></p>
<p><span style="font-family:"Arial","sans-serif"">Making a Difference in the Fight Against Breast Cancer</span><u></u><u></u></p>
<p> <u></u><u></u></p>
<p> <u></u><u></u></p>
<p><strong><span style="font-family:"Arial","sans-serif"">How and Why I Should Support Bottled Water!</span></strong><b><span style="font-family:"Arial","sans-serif""><br>
</span></b><span style="font-family:"Arial","sans-serif"">Do not relinquish your right to choose bottled water as a healthy alternative to beverages that contain sugar, calories, etc. Your support of bottled water will make a difference! Your signatures count!
Go to <a href="http://www.bottledwatermatters.org/luv-bottledwater-iframe/dswaters" target="_blank">
http://www.bottledwatermatters.org/luv-bottledwater-iframe/dswaters</a> and sign a petition to support your right to always choose bottled water. Help fight federal and state issues, such as bottle deposits (or taxes) and organizations that want to ban the
sale of bottled water. Support community curbside recycling programs. Support bottled water as a healthy way to maintain proper hydration. Our goal is 50,000 signatures. Share this petition with your friends and family today!</span><u></u><u></u></p>
<p> <u></u><u></u></p>
<p><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">---------------------------------<br>
CONFIDENTIALITY NOTICE: This e-mail may contain privileged or confidential information and is for the sole use of the intended recipient(s). If you are not the intended recipient, any disclosure, copying, distribution, or use of the contents of this information
is prohibited and may be unlawful. If you have received this electronic transmission in error, please reply immediately to the sender that you have received the message in error, and delete it. Thank you.<br>
----------------------------------</span><u></u><u></u></p>
<p> <u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org" target="_blank">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <br>
David Tomaschik<br>
OpenPGP: 0x5DEA789B<br>
<a href="http://systemoverlord.com" target="_blank">http://systemoverlord.com</a><br>
<a href="mailto:david@systemoverlord.com" target="_blank">david@systemoverlord.com</a>
<u></u><u></u></p>
</div>
</div></div></div>
</div>
<br>_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
<br></blockquote></div><br></div>