CRS - Restore after server failure

About

This chapter describes how to restore YSoft SafeQ CRS after the failure. This scenario shall be followed in case of any kind of CRS failure (corrupted hardware, corrupted SQL server, missing database files, corrupted CRS binary files...).

The article consists of:

Requirements

Kindly prepare all the required data before starting with the master node (1st installed node) recovery:

  • CRS databases backups

  • CRS configuration backup

  • CRS installation files

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/lightbulb_on.svg See article CRS - Backup for more information about database and configuration files backup.

Step 1 - restore hardware after server failure (optional)

Replace all corrupted hardware.

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/lightbulb_on.svg Skip this step in case that hardware was not corrupted.

Step 2 - restore software after server failure (optional)

Use this step in case the operating system or MS SQL has been corrupted

1

Install the operating system.

The operating system must be supported for the particular version of MS SQL server that you are going to deploy. See documentation of your MS SQL server for more details.

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/warning.svg It is important to have the same IP address like on the previous server which is corrupted.

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/lightbulb_on.svg Skip this step in case the Microsoft Windows was not corrupted.

2

Install and configure Microsoft SQL Server for YSoft SafeQ CRS.

It is advised to install the same version/edition of MS SQL server which was used before the failure to avoid the database compatibility issues. The details for the MS SQL server installation can be found in following articles:

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/lightbulb_on.svg Skip this step in case the MS SQL Server was not corrupted.

 

3

Enable replication as follows:  

  1. Start Microsoft SQL Management Studio and connect to the Database Engine as the user sa.

  2. Right-click on the Replication to display the context menu; then select Configure Distribution. (If this menu is not available and you see Publisher properties instead, this step has been probably already done and you can proceed to next step).

  3. In the window that opens, click Next; then Finish; then Finish again.

  4. Replication is now enabled.

Step 3 - clean up the server before CRS re-installation

It is necessary to clean up (remove) all the components described below before the YSoft SafeQ CRS can be restored.

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/lightbulb_on.svg You can skip this step in the case of clean installation of Operating system and MS SQL server.

 

1

Make sure the MS SQL server is properly installed, including Reporting services.

Open the SQL Server Configuration Manager (Start > Programs > Microsoft SQL Server 20xx > Configuration tools -> SQL Server Configuration Manager) and verify required services are running.

 

2

Make sure the YSoft SafeQ CRS service is removed

Open a Windows Service (run -> services.msc). If the YSoft SafeQ CRS service exists:

  1. Stop the service

  2. Delete the service using command prompt:

sc delete YSoftSafeQCRS


3

Make sure that CRS folder is deleted

Check disk drive where CRS server was installed. In the case that SafeQCRS folder exists, delete it.

 

4

Make sure that no CRS databases exist on the MS SQL server

  1. Connect to the Database Engine using SQL Server Management Studio.

  2. If yBoxDBREP, yBoxDB or yBoxDBCDC databases exist, delete them.

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/lightbulb_on.svg Hints DB for removal:

If the deletion of databases fails, take the database offline (right-click the database -> Tasks > Take Offline), delete the database file (location of related mfd and ldf file can be found by right-clicking the database -> Properties -> tab Files -> information under "Database files" text) and then delete the database via SQL Server Management Studio.

In the database cannot be taken offline:

  1. find the session which is currently blocking the database. SQL command:
    USE [master]
    select d.name, d.dbid, spid, login_time, nt_domain, nt_username, loginame from sysprocesses p inner join sysdatabases d on p.dbid = d.dbid where d.name = '<databasename>'
    GO

  2. kill the blocking session. SQL command:
    USE [master]
    kill <spid number from previous query>
    GO

  3. disable single-user session for this DB . SQL command:
    USE [master]
    exec sp_dboption '<databasename>', 'single user', 'FALSE'
    GO

  4. it shall be now possible to take DB offline and delete the database files. For more information see the documentation of your MS SQL server.

5

Make sure that packages for cubes processing are deleted

  1. Connect to the Integration services using SQL Server Management Studio.

  2. Following packages must not be present in Running packages or Stored Packages\MSDB:

    • PackageBuildSafeq

    • PackageBuildSafeq_full

In case the packages are present, delete them.

 

6

Make sure that analysis services does not contain SafeQCDC

Connect to Analysis Services using SQL Server Management Studio. If the database SafeQCDC database exists, delete it.

 

Step 4 - restore the YSoft SafeQ CRS

1

Restore the YSoft SafeQ CRS databases (according to CRS - Database Restore) on MS SQL server :

  • yBoxDB

  • yBoxDBCDC

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/warning.svg Be sure that:

  • yBoxDBREP database does not exist on MS SQL server.

  • publication SafeQREP does not exist (MS SQL Server Management studio -> Database engine -> Replication -> Local Publications). Should it exist, delete it.

2

Configure the CRS server as described in CRS configuration, make sure the following attribute in DeploymentConfig\crs\env.ini is set:

Database instance creation type 
databaseInstanceType = existing-db


3

Run the install.bat to begin the CRS installation: InstallPackages\install.bat.

The command prompt opens. Enter the environment name, including the path to the currently installed node:
Example: crs\TEST\node1

images/download/attachments/6424751/crs1.PNG

4

You can see the installation progress in the command line window. The installation shall finish with the message INSTALL PROCESS FINISHED SUCCESSFULLY. Then you can close the installation window.
images/download/attachments/6424751/crs2.PNG

5

Verify that YSoft SafeQ CRS service is running (using service.msc)

If it is stopped start it manually.images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/lightbulb_on.svg If the service cannot be started, more information can be found in <SafeQCRS_DIR>\logs\crs.log

 

Step 5 - verification of the functionality

1

Verification of connectivity to the CRS server

It must be possible to access the statistical data on the CRS server. The date/time in the statistics shall be equal to the date/time when the CRS backup has been performed. Please verify that data are available.

1a. Verification of access via Analytic Services

1b. Verification of access via Reporting Services (optional)

  • perform this step only if Reporting Services were installed

  • follow the description from article Working with YSoft SafeQ CRS reports to see how to obtain statistics from the CRS using Reporting Services

2

Verification of data synchronization from CML to CRS

Send the new data to the CRS Server

  • Open YSoft SafeQ Web interface of the CML server that is sending statistics to the CRS.

  • Open tab Reports -> CRS Reports -> Actions -> click Send statistics to the CRS

  • Wait till the Sending process finished message is shown. At least some rows has to be transferred and sending must not fail.

images/download/attachments/21955765/StatisticsSync.png

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/lightbulb_on.svg In case there are no new statistical data on the CML, the sending of statistics may still finish properly but the "Transferred rows" will contain 0. In such a case do the following:

  • make a testing print

  • verify that the print has been accounted in job history

  • wait 1 hour for the statistical data to be processed on the CML level

  • try to send the statistical data again


3

Verification of statistical data processing

3a. Check that CRS data processing is not in progress

Verify that there is no cube processing running at the moment on the CRS server. If the package is running, wait for it to finish.

  • Using SQL Server Management Studio connect to the Integration services

  • Make sure that no package is being shown in Running Packages. The name of package may vary.

images/download/attachments/21955765/RunningPackage.png

3b. Restart CRS service

Restart the YSoftSafeQCRS service (via services.msc) on the CRS server. This will force processing of data in the cube and cube processing package will start.

 

3c. Wait for cube processing to finish

Wait till the cube processing is finished. The cube processing is finished when there is no package running any more. To verify when the package was processed follow these steps:

  • Using SQL Server Management Studio connect to the Integration services

  • Expand Databases -> SafeQCDC -> Cubes -> right-click Base Stats -> select Properties

  • Status must be Processed and the time at Last processed is subsequent to the time of CRS service restart.

images/download/attachments/21955765/CubeProcessed.png images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/lightbulb_on.svg Processing may take a few hours depending on the size of the database. More details about processing can be found in the <SafeQCRS_DIR>\logs\crs.log (thread CSDManagerCRS).

 

3d. Verify the statistical data

Verify the CRS report includes the data from the last synchronization. The data in the CRS shall have the date/time equal to the last time of synchronization (as referred in point 2).