Data Migration

Migrate Elasticsearch Data

Pre-requisites

System requirements

• For optimal performance, 10ms or lower latency is recommended, and it should not exceed 100ms.
• 50% of the server's RAM should be kept free for off-heap utilization of Elasticsearch for optimal performance.

Overview of Elasticsearch (ES) data paths

ES directories

• ES\repo folder contains temporary files for ES archives
• ES\data folder contains data
• ES\archive folder contains ES archives
• ES\repo, ES\data and ES\archive should never point to the same folder

Examples:

For remote network path use the following format:

• path.data : ["//remote machine name/shared folder/data"]
• path.repo : ["//remote machine name/shared folder/repo"]

For Windows local storage, use the following format:

• path.data : ["C:\\ManageEngine\\EventLog Analyzer\\ES\\data"]
• path.repo : ["C:\\ManageEngine\\EventLog Analyzer\\ES\\repo"]

For Linux local storage, use the following format:

• path.data : ["/opt/ManageEngine/EventLog Analyzer/ES/data"]
• path.repo : ["/opt/ManageEngine/EventLog Analyzer/ES/repo"]

Steps to migrate Elasticsearch data to a new location

Case 1: EventLog Analyzer is integrated with Log360 and is installed with Log360 installer (Bundled)

In this case, EventLog Analyzer uses a common Elasticsearch that is shared with the other modules.

Figure 1: Details of the Elasticsearch node running location

Here are the steps to migrate data for EventLog Analyzer bundled with Log360:

1. Shutdown EventLog Analyzer and Log360.
2. Shutdown common ES.
• Open Command Prompt as the Administrator in <ManageEngine>\elasticsearch\ES\bin
• Run stopES.bat
3. Copy the data directory from <ManageEngine>\elasticsearch\ES\data to the new location.
4. Navigate to <ManageEngine>\elasticsearch\ES\config\elasticsearch.yml, and update path.data to include the new data location. (Refer to Figure 2)
5. Update path.repo in <ManageEngine>\elasticsearch\ES\config\elasticsearch.yml to the new repository location (parallel to the new data path).
6. Start the Log360 application along with its EventLog Analyzer module.

Figure 2: Update path.data and path.repo to new location

Case 2: EventLog Analyzer as a standalone setup (Not integrated with Log360)

Here are the steps to migrate data for standalone EventLog Analyzer:

1. Shutdown EventLog Analyzer.
2. Copy the data directory from <ManageEngine>\EventLog Analyzer\ES\data to the new location.
3. Create a folder with the name archive (parallel to the new data directory).
4. Move the files from <ManageEngine>\<EventLog Analyzer>\ES\archive folder to the new folder named archive.
5. Navigate to <ManageEngine>\<EventLog Analyzer>\ES\config\elasticsearch.yml, and update path.data to include the new data location. (Refer to Figure 2)
6. Update path.repo in <ManageEngine>\<EventLog Analyzer>\ES\config\elasticsearch.yml to the new repository location (parallel to the new data path).
7. Start EventLog Analyzer.

Case 3: EventLog Analyzer is manually integrated into Log360

Figure 3: Details of the running location Elasticsearch nodes

Here are the steps to migrate data for EventLog Analyzer integrated with Log360:

1. Shutdown EventLog Analyzer and Log360.
2. Shutdown common ES.
• Open Command Prompt as the Administrator in <ManageEngine>\elasticsearch\ES\bin
• Run stopES.bat

There are two running Elasticsearch nodes for which we need to migrate data:

A. Migrating common ES data

1. Copy the data directory from <ManageEngine>\elasticsearch\ES\data to the new location.
2. Navigate to <ManageEngine>\elasticsearch\ES\config\elasticsearch.yml, and update path.data to include the new location. (Refer to Figure 2)
3. Update path.repo in <ManageEngine>\elasticsearch\ES\config\elasticsearch.yml to include the new repository location (parallel to path.data).

B. Migrating local ES data

1. Copy the data directory from <ManageEngine>\EventLog Analyzer\ES\data to the new location.
2. Navigate to <ManageEngine>\<Eventlog>\ES\config\elasticsearch.yml, update path.data to include the new location. (Refer to Figure 2)
3. Update path.repo in <ManageEngine>\<Eventlog home>\ES\config\elasticsearch.yml to the same repository location as that of common ES.
4. Create a folder with the name archive (parallel to the new data directory).
5. Move the files from <ManageEngine>\<Eventlog>\ES\data to the new location.
6. Move the files from <ManageEngine>\<Eventlog>\ES\archive folder to the new folder named archive.
7. Start Log360 and EventLog Analyzer.

Migrate Archive Data

Pre-requisites

System requirements

Steps to migrate the old archive data

Steps to change the location of the archives in the database

Here are steps to change the location of the archives in the database:

1. If the newly created archives should be stored in the new location
   Go to Settings > Admin Settings > Data Storage > Archives > Settings

Update the zip location to the new location. If it is required to be stored in the old location, then this step can be skipped. (Refer to Figure 1)

Figure 1: Update archive location

2. Manually move the old archives from the old location to the new location.
3. Now the location has to be updated in the database for each archive. (Refer to Figure 2)
   Go to Settings > Admin Settings > Data Storage > Archives > More in the top right corner Update path.

Figure 2: Archive data - Update paths

4. Select the old archive location in the dropdown and enter the new location where the archives are moved.
5. Once all the archive locations are updated, click on the Refresh icon in the top right corner to update the status of the archives. (Refer to Figure 3)

Figure 3: Update status of archives