Receive DB Slow Query Logs in Your Inbox

Want to wake up to important database and syslog messages with your bagel and coffee? Here’s how. It’s free and takes about 5 minutes.

Most of us run a database somewhere on our SoftLayer servers. Whether it’s MySQL, PostgreSQL, SQL Server, or another relational or NoSQL sibling, a responsive data store is critical to happy users. That’s why databases send slow queries to a log file. It’s much better than no logging at all, but as an engineer, I’d wanted more. I wanted to:

• View all my query logs in one place, without SSHing to each server for tail and grep. My workload shouldn’t scale linearly as I add systems
• Share log visibility with employees who don’t have server access or command-line knowledge (and email links to specific log messages to my developers and DBAs)
• Receive log messages in my inbox – or send them to my team or monitoring service – when I know they need attention
• Examine logs for related HTTP requests, daemon output, API invocations, and other parts of our stack — I can troubleshoot faster with start-to-finish logs on a single screen.

That’s where Papertrail was born. We built Papertrail to make log aggregation and log management effortless and usable. It’s the hosted log management service that we wanted as developers, systems engineers and tech entrepreneurs.

We know the hesitation you might have when approaching this kind of service, so our goal was to enable users to have Papertrail deliver those SQL slow query logs – or any other logs – to your inbox every morning for free:

1. Register a Papertrail log Repository
Hit https://papertrailapp.com/plans and sign up. You’ll land on a welcome page. From there, click the “Add System” link, and on the form shown, type the IP of your DB server and its name (like “db1″).

Papertrail will display a remote syslog destination for your logs and easy instructions for sending app log files and syslog. The steps below are customized for MySQL’s slow query log.

2. Send MySQL logs
Install a log sender. To install a tiny standalone binary that sends log file contents to Papertrail as they occur without any other system-wide configuration changes, with Ruby you can run the “gem” binary: sudo gem install remote_syslog

Next, locate your MySQL or other database slow query log file. Usually these are in the directory /var/mysql/log/ or /var/log/mysql/, usually with a filename containing “slow”. An example slow query log path is: /var/log/mysql/mysql-slow.log

If you don’t see one, try running “locate mysql | grep slow.log” (or to configure a slow query log, head here).

Next, tell remote_syslog the path to that file. Edit /etc/log_files.yml and add a single line:
files: [/var/log/mysql/mysql-slow.log]

Use the path to your slow query log (Example). Finally, run remote_syslog:
sudo remote_syslog

3. Set up Nightly Email
Now that you’ve completed those first two steps, you’re configured to send those slow query logs to Papertrail as soon as they happen. Hit Papertrail’s events viewer to watch them roll in. Because only slow queries are being logged, you may not see any events immediately.

To receive these messages in email, search Papertrail’s events viewer for ‘mysql’. Then click “Save Search” and give this search a name (like “MySQL slow queries”). Click the Dashboard link and you’ll see that search as a new choice. Click the Edit link next to it, and you’ll be prompted for the email address(es) which should receive these in email every night.

That’s it. Congrats! If you were hoping for more steps, I’d recommend that you get yourself a Doppelbock or porter as Step 4.

You have the ability to give co-workers access to these logs in Papertrail, and because Papertrail doesn’t charge a per-system fee, you can add other systems at will. With your SQL logs done, it only takes a few more minutes to aggregate Web server request logs and OS syslog results from your dedicated and CloudLayer systems (try “Add System” or “Quick Start”).

Enjoy!

- Troy Davis, Papertrail