How to back up indexed data or change default database location in Splunk

In Splunk, data is indexed into database directories called buckets. Indexed data is transferred from one type of bucket to another as it ages. Below types of buckets are listed:

Bucket type

Information

Default location

Hot bucket

Currently being written to

(format: hot_v1_<ID>)

<Splunk directory>\var\lib\splunk\defaultdb\db

Warm bucket

Rolled from hot

(format: db_<newest_time>_ <oldest_time>_<localid>)

<Splunk directory>\var\lib\splunk\defaultdb\db

Cold bucket

Rolled from warm

<Splunk directory>\var\lib\splunk\defaultdb\colddb

Frozen bucket

Gets deleted or archieved

Thawed bucket

Archieved and later thawed data

<Splunk directory>\var\lib\splunk\defaultdb\thaweddb

Firstly, data is written to a hot bucket. When hot bucket reaches to a specified size/age or whenever Splunkd service gets restarted, data in hot bucket moved/rolled into a warm bucket. When number of warm buckets reaches to a value (default: 300), data in oldest warm bucket is moved/rolled into a cold bucket.

To backup indexed data

  • Simply copy the following directories and their entire content to another location:
    • <Splunk directory>\var\lib\splunk\defaultdb\db
      <Splunk directory>\var\lib\splunk\defaultdb\colddb
      <Splunk directory>\var\lib\splunk\defaultdb\thaweddb
      (* default splunk directory: “C:\Program Files\Splunk”)

Note: If there is a hot bucket (in format “hot_v1_<ID>”), carry out one of the following steps to roll it to a warm bucket:

  • Simply restart the Splunkd service in command line (preffered)
    • net stop/start splunkd
  • Or manually roll hot bucket to warm in command line (not preffered)
    • splunk _internal call /data/indexes/<index_name>/roll-hot-buckets –auth <admin_username>:<admin_password>

To change default database location

  • Stop Splunkd service in command line
    • net stop splunkd
  • Copy indexes.conf file under “<Splunk directory>\etc\system\default” to “<Splunk directory>\etc\system\local”
  • Edit indexes.conf file under “<Splunk directory>\etc\system\local” as below:
    • Locate the “index definitions” section
    • Change the location of each database under “[main]” tag as below:
      • homePath: <New database path>\db
        coldPath: <New database path>\colddb
        thawedPath: <New database path>\thaweddb
        (* example new database path: “C:\Splunk DB\project 1”)
  • Start Splunkd service
    • net start splunkd

This entry was posted in Data Analysis and tagged , , , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *