[ale] Log parsing/alerting tool recomendations

[Scott Bragg]

It's a bit on the expensive side, but for
all-purpose-do-everything-imaginable log *parsing* and *reporting* tool,
you can't go wrong with Splunk.


On Thu, May 30, 2013 at 8:39 AM, JD <jdp at algoloma.com> wrote:

>  I've worked on both sides of this issue and like to think the places
> where I worked did logging better than anyone else.
>
> The exact format of the logs isn't as important as having the necessary
> data to exactly determine where any problem lies.
> 1. Log files should be all that any support person needs to determine an
> issue with the program. Developers **should NEVER** need to login on any
> production server.
> 2. Logs should have multiple levels which can dynamically be turned up or
> down WHILE the program is running. No restart needed.
> 3. Timestamp is mandatory.
> 4. ERROR|WARNING|INFO to help with grepping later by knowledgeable people.
> 5. Program name (including unique "build" identifier, module, function,
> method, and line number are highly helpful
> 6. A program/script provided by the vendor to gather any necessary
> troubleshooting information is MANDATORY. Telling a low skilled person to
> gather log-A and config-A thru Z files and syslog-B and uname-a and .... is
> not good. Have the vendor create AND provide a script that captures
> whatever information they will need to troubleshoot.
> 7. Startup and shutdown would be nice "INFO" entries for the program.
> 8. Use syslog when possible so the local admin can have control over the
> final target of the logs. It also means that using a tool like cacti any/or
> splunk is easier.
>
> For fat-clients, access to a stack trace that an end-user can copy/paste
> into an email is extremely helpful too. It is amazing what the __MODULE__
> and __LINE__ macros can do for pinpointing issues.  Being able to tell a
> client exactly what issue they've hit in about 3 minutes on the phone saves
> you AND the customer time. Until a fix is available, the client can avoid
> that area of the code and you know exactly which check is failing (or have
> a 95%+ likely location).
>
>
>
> On 05/29/2013 02:49 PM, Jim Kinney wrote:
>
>    check this site for an idea:
>
> http://www.w3.org/Daemon/User/Config/Logging.html
>
>  I find apache logs very easy to parse with many tools.
>
>  Another way to look at this is from the viewpoint of someone having to
> understand why the apps are not working as they think they should. What
> type of data would help?  At that point, the data type will usually dictate
> an output format. And sysadmins _LOVE_ error messages like:
>
>  application foo received 0x120BAF02 at 0x33BD000001A1. Is this OK?
>
>  That is useless! Saw that (different addresses) during a Debian install
> once. I thought my head would explode.
>
>  I've seen many java applications (java is a great drink and a country
> I've never visited. It's a crappy language that should not be taught - grr)
> that split up logs into sort of a user/admin general process, admin error,
> and application error tracking. Each had a deeper level of details with
> very long time stamps.
>
>
> On Tue, May 28, 2013 at 6:34 PM, Robert L. Harris <
> robert.l.harris at gmail.com> wrote:
>
>>
>>    I'm working with a number of developers trying to create a logging
>> standard for some apps and devices my company is developing.  Most of them
>> are linux based and running syslog-ng so we have some flexibility and can
>> standardize.  The big concern though is coming up with a format for the
>> logs for the tools we will (may) be using to parse the data.  Personally I
>> like the idea of using cmd line and piping unix utils.
>>
>>    A recommendation was thrown out though to ask about how others are
>> parsing system and application logs to look for issues, tracking, etc and
>> what kinds of input they take (json, xlm, .log, etc).  Anyone have any
>> tools you're using that are just incredible and what kinds of input they
>> can work with?
>>
>>  Robert
>>
>>
>>  --
>> :wq!
>>
>
>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ale.org/pipermail/ale/attachments/20130530/de7da0d0/attachment.html>