Add support for automatic removal of old logs (google#432)

* Add support for automatic removal of old logs

GetOverdueLogNames(string log_directory, int days) will check all
filenames under log_directory, and return a list of files whose last modified time is
over the given days (calculated using difftime()).

So that we can easily for unlink all files stored in the returned vector.

* Replaced the lines that require C++11

* embed dirent.h in project

* Add support for automatic removal of old logs

In this commit, at the end of LogFileObject::Write,
it will perform clean up for old logs.

It uses GetLoggingDirectories() and for each file in each
directory, it will check if a file is a log file produced by glog.
If it is, and it is last modified 3 days ago, then it will unlink()
this file. (It will only remove the project's own log files,
it won't remove the logs from other projects.)

Currently it is hardcoded to 3 days, I'll see if this can be
implemented in a more flexible manner.

* Implement old log cleaner

The log cleaner can be enabled and disabled at any given time.
By default, the log cleaner is disabled.

For example, this will enable the log cleaner and delete
the log files whose last modified time is >= x days
google::EnableLogCleaner(x days);

To disable it, simply call
google::DisableLogCleaner();

Please note that it will only clean up the logs produced for
its own project, the log files from other project will be untouched.

* logging: log_cleaner: Use blackslash for windows dir delim

* logging: log_cleaner: remove the range-based loops

Also replaced the hardcoded overdue days with the correct variable.

* Add Marco Wang to AUTHORS and CONTRIBUTORS

* logging: log_cleaner: Remove redundant filename stripping

Previously the full path to a file is passed into IsGlogLog(),
and then std::string::erase() is used to get the filename part.
If a directory name contains '.', then this function will be unreliable.

Now only the filename it self is passed into IsGlogLog(),
so this problem will be eradicated.

* logging: log_cleaner: improve readability

* Add google::EnableLogCleaner() to windows logging.h

* logging: log_cleaner: Remove perror message

* logging: IsGlogLog: match filename keyword by keyword

Splitting a filename into tokens by '.' causes problems
if the executable's filename contains a dot.

Filename should be matched keyword by keyword in the following
order:
1. program name
2. hostname
3. username
4. "log"

AUTHORS

@@ -15,6 +15,7 @@ Andy Ying <[email protected]>
 Brian Silverman <[email protected]>
 Google Inc.
 Guillaume Dumont <[email protected]>
+Marco Wang <[email protected]>
 Michael Tanner <[email protected]>
 MiniLight <[email protected]>
 romange <[email protected]>

CONTRIBUTORS

@@ -33,6 +33,7 @@ Håkan L. S. Younes <[email protected]>
 Ivan Penkov <[email protected]>
 Jacob Trimble <[email protected]>
 Jim Ray <[email protected]>
+Marco Wang <[email protected]>
 Michael Tanner <[email protected]>
 MiniLight <[email protected]>
 Peter Collingbourne <[email protected]>

src/glog/logging.h.in

@@ -525,6 +525,11 @@ GOOGLE_GLOG_DLL_DECL void ShutdownGoogleLogging();
 // Install a function which will be called after LOG(FATAL).
 GOOGLE_GLOG_DLL_DECL void InstallFailureFunction(void (*fail_func)());
 
+// Enable/Disable old log cleaner.
+GOOGLE_GLOG_DLL_DECL void EnableLogCleaner(int overdue_days);
+GOOGLE_GLOG_DLL_DECL void DisableLogCleaner();
+
+
 class LogSink; // defined below
 
 // If a non-NULL sink pointer is given, we push this message to that sink.

src/logging.cc

@@ -44,6 +44,7 @@
 #ifdef HAVE_SYS_UTSNAME_H
 # include <sys/utsname.h> // For uname.
 #endif
+#include <time.h>
 #include <fcntl.h>
 #include <cstdio>
 #include <iostream>
@@ -58,6 +59,11 @@
 #include <vector>
 #include <errno.h> // for errno
 #include <sstream>
+#ifdef OS_WINDOWS
+#include "windows/dirent.h"
+#else
+#include <dirent.h> // for automatic removal of old logs
+#endif
 #include "base/commandlineflags.h" // to get the program name
 #include "glog/logging.h"
 #include "glog/raw_logging.h"
@@ -833,6 +839,86 @@ void LogDestination::DeleteLogDestinations() {
  sinks_ = NULL;
 }
 
+namespace {
+
+bool IsGlogLog(const string& filename) {
+ // Check if filename matches the pattern of a glog file:
+ // "<program name>.<hostname>.<user name>.log...".
+ const int kKeywordCount = 4;
+ std::string keywords[kKeywordCount] = {
+ glog_internal_namespace_::ProgramInvocationShortName(),
+ LogDestination::hostname(),
+ MyUserName(),
+ "log"
+ };
+
+ int start_pos = 0;
+ for (int i = 0; i < kKeywordCount; i++) {
+ if (filename.find(keywords[i], start_pos) == filename.npos) {
+ return false;
+ }
+ start_pos += keywords[i].size() + 1;
+ }
+ return true;
+}
+
+bool LastModifiedOver(const string& filepath, int days) {
+ // Try to get the last modified time of this file.
+ struct stat file_stat;
+
+ if (stat(filepath.c_str(), &file_stat) == 0) {
+ // A day is 86400 seconds, so 7 days is 86400 * 7 = 604800 seconds.
+ time_t last_modified_time = file_stat.st_mtime;
+ time_t current_time = time(NULL);
+ return difftime(current_time, last_modified_time) > days * 86400;
+ }
+
+ // If failed to get file stat, don't return true!
+ return false;
+}
+
+vector<string> GetOverdueLogNames(string log_directory, int days) {
+ // The names of overdue logs.
+ vector<string> overdue_log_names;
+
+ // Try to get all files within log_directory.
+ DIR *dir;
+ struct dirent *ent;
+
+ char dir_delim = '/';
+#ifdef OS_WINDOWS
+ dir_delim = '\\';
+#endif
+
+ // If log_directory doesn't end with a slash, append a slash to it.
+ if (log_directory.at(log_directory.size() - 1) != dir_delim) {
+ log_directory += dir_delim;
+ }
+
+ if ((dir=opendir(log_directory.c_str()))) {
+ while ((ent=readdir(dir))) {
+ if (!strcmp(ent->d_name, ".") || !strcmp(ent->d_name, "..")) {
+ continue;
+ }
+ string filepath = log_directory + ent->d_name;
+ if (IsGlogLog(ent->d_name) && LastModifiedOver(filepath, days)) {
+ overdue_log_names.push_back(filepath);
+ }
+ }
+ closedir(dir);
+ }
+
+ return overdue_log_names;
+}
+
+// Is log_cleaner enabled?
+// This option can be enabled by calling google::EnableLogCleaner(days)
+bool log_cleaner_enabled_;
+int log_cleaner_overdue_days_ = 7;
+
+} // namespace
+
+
 namespace {
 
 LogFileObject::LogFileObject(LogSeverity severity,
@@ -1125,7 +1211,7 @@ void LogFileObject::Write(bool force_flush,
  file_length_ += header_len;
  bytes_since_flush_ += header_len;
  }
-
+ 
  // Write to LOG file
  if ( !stop_writing ) {
  // fwrite() doesn't return an error when the disk is full, for
@@ -1175,12 +1261,21 @@ void LogFileObject::Write(bool force_flush,
  }
  }
 #endif
+ // Perform clean up for old logs
+ if (log_cleaner_enabled_) {
+ const vector<string>& dirs = GetLoggingDirectories();
+ for (size_t i = 0; i < dirs.size(); i++) {
+ vector<string> logs = GetOverdueLogNames(dirs[i], log_cleaner_overdue_days_);
+ for (size_t j = 0; j < logs.size(); j++) {
+ static_cast<void>(unlink(logs[j].c_str()));
+ }
+ }
+ }
  }
 }
 
 } // namespace
 
-
 // Static log data space to avoid alloc failures in a LOG(FATAL)
 //
 // Since multiple threads may call LOG(FATAL), and we want to preserve
@@ -2216,4 +2311,18 @@ void ShutdownGoogleLogging() {
  logging_directories_list = NULL;
 }
 
+void EnableLogCleaner(int overdue_days) {
+ log_cleaner_enabled_ = true;
+
+ // Setting overdue_days to 0 day should not be allowed!
+ // Since all logs will be deleted immediately, which will cause troubles.
+ if (overdue_days > 0) {
+ log_cleaner_overdue_days_ = overdue_days;
+ }
+}
+
+void DisableLogCleaner() {
+ log_cleaner_overdue_days_ = false;
+}
+
 _END_GOOGLE_NAMESPACE_