<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Trebuchet MS";
panose-1:2 11 6 3 2 2 2 2 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>As the number of directories with Fedora content has exploded, in large part due to splitting the 20k files into separate [a]/ [b]/ [c]/ directories, the MirrorManager crawler has gotten slower too. 90 minutes and even 2 hours isn’t sufficient anymore, especially for sites that are complete mirrors. To try to alleviate this, I’ve bumped up the MM crawler timeout to 3 hours, but that’s not a great solution.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>If your mirror offers an rsync target, and you list your rsync URL in MirrorManager for each category, the crawler will now use that (thanks to Adrian Reber). This can take crawls down from >2 hours to ~25 minutes, most of which time is spent hammering the MM database instead of hitting your mirror with a ton of HTTP HEAD requests.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Please consider adding an rsync module for each Category of content you carry, and add the URLs into MirrorManager.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks,<o:p></o:p></p><p class=MsoNormal>Matt<o:p></o:p></p><p class=MsoNormal><b><span style='font-size:8.0pt;font-family:"Trebuchet MS","sans-serif";color:#444444'><o:p> </o:p></span></b></p><p class=MsoNormal><b><span style='font-size:8.0pt;font-family:"Trebuchet MS","sans-serif";color:#444444'>--<o:p></o:p></span></b></p><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#444444'>Matt Domsch<o:p></o:p></span></b></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Trebuchet MS","sans-serif";color:#444444'>Distinguished Engineer, Technology Strategist<o:p></o:p></span></p><p class=MsoNormal><b><span style='font-size:8.0pt;font-family:"Trebuchet MS","sans-serif";color:#0085C3'>Dell</span></b><span style='font-size:8.0pt;font-family:"Trebuchet MS","sans-serif";color:#444444'> | Office of the CTO<b><o:p></o:p></b></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>