Continuing my series of making your home more like a serious business environment, this time I want to talk about log collection. We touched on this a little in “Doing IT at Home: Ticketing and Monitoring” as Spiceworks, which I mentioned there, does some amount of Windows event collection. That was a very light treatment of the subject, however, and a serious “Do IT at Homer” is going to want something more robust and enterprise class.
Enterprise log collection, searching and reporting has really moved from a niche to a core IT tool over the past decade with the charge lead by the ubiquitous Splunk. There are many products that potentially fit into this category with varying degrees of features and robustness. Traditional “old school” logging systems tend to fall into the categories of Windows event collectors like Spiceworks, Windows Event Collector and ManageEngine EventLog Analyzer. In the UNIX world and in networking hardware worlds we tend to work with syslog compatible systems like Rsyslog and Solarwinds Kiwi Syslog Server. But these products are pretty limited, being very focused on limited platforms and are often quite expensive (Kiwi) or lack a good user experience (Rsyslog.) In exploring for your home lab it may make sense to play with some of these products. But for really taking your logging to the next level we are going to need to looks at vastly more robust platforms that address all of these data sources and more, are extensible and are designed around not only collecting data but making it searchable and displayable and, hopefully, usable by more than just the hard core home system administrator.
Leading the charge for this new breed of log collection systems is Splunk. Splunk is primarily an on-premise, proprietary software package but is available with a “Free” option that is generally perfect for a home IT enthusiast. The Free edition limits the volume of daily log ingest and does not support multiple users which is unlikely to be a real stumbling block for home use. The ingest limit is currently 500MB per day which is an incredible volume of logs . Splunk understands that their paying customers will only ever be larger shops with large log volumes so giving their product away for free for small shops and personal users actually helps their bottom line by encouraging broader experience with and knowledge of their product. Splunk is relatively complex and will take some effort to set up but is extremely powerful and featureful.
Splunk is hardly the only on premise log handling game in town. In the open source realm there is a flurry of activity around log collection and reporting, mostly built atop the Elasticsearch NoSQL data platform and the key one is known as the “ELK” stack referring to the three principle components: Elasticsearch, Logstash and Kibana. A common alternative is to keep the stack but to replace Kibana, the data analytics interface, with Graylog2 which is also open source. The ELK or similar stacks provide very “Splunk like” functionality without the Splunk limitations. Splunk is certainly the more popular choice for enterprises today but ELK is making significant inroads in mindshare and is to be seen most often in more innovative companies such as technology startups, research firms and large hosting services (Dreamhost is a notable sponsor.)
Tackling an on premise log management project will provide a great excuse for all of that extra hardware lying around the house and will provide deeper systems administration experience as there is more “server” to be managed and maintained. Unlike in a business, when doing IT at Home there are significant benefits to technology sprawl and intentionally taking on the more difficult path. We are actively seeking challenges and meaningful systems to be run at home that produce real value and log analysis is a great place to add value by leveraging data that your network is already creating and providing it to you in a way that makes you better able to anticipate problems before they occur, track down issues after the fact and dramatically increase security – knowing what is going on on your network and in your devices has a lot of value and manually parsing Windows Events and UNIX syslogs is boring and error prone. Looking at graphical data is more effective and reliable. And logging platforms can send alerts based on logging events as well.
On premise log management is not the only option. Log management is also available in a Software as a Service business model with two really key players, Splunk – by way of their “Splunk Storm” service and the market leader, Loggly. Both of these vendors offer completely free, capacity limited versions of their hosted products which are far more than any home IT user will need. These services allow you to get up and running with enterprise log management in a matter of minutes without any investment, neither in time nor money nor hardware. If your goals are less around learning system administration and more purely around focusing on good log management or you simply lack the racks of hardware at home that precipitate intentionally creating “extra” IT services in the house then hosted log management is very likely the right choice for you. For those focused on development, network administration or other areas of IT this is likely the more useful option. Loggly, especially, is easy for anyone to sign up and start sending log data and is the leading hosted log management product today.
The larger and more active that your home network becomes the more valuable good log collection and management becomes. Logs provide deep insight into your network and a good log management solution will not only provide you with a high level view of your data but will also be useful in replacing the traditional views of your live log data. Working from an attractive web interface is generally far more effective than manually scouring logs, even when looking at current events. And some log management solutions will also provide good facilities for long term log retention which is often lacking in non-centralized solutions.
Good log management is rapidly becoming more and more important and an expected service in business. Five years ago it was common to see even very large enterprises not yet adopting these kinds of tools. Today it is assumed that any company of any reasonable size will have a solid, mature logging solution, almost certainly Splunk, in place and with the increasingly lower and lower barriers to entry from ELK and Loggly we see enterprise logging working its way into smaller and smaller firms. Logging at home is an excellent way to enhance your personal portfolio, extend your knowledge and skillbase and build up your resume. Get logging today!