Updating YSoft CML cluster

1. Update procedure at glance

Step No

Action to perform

Comment

User impact

1

Shut down the whole environment. Leave only YSoft SafeQ CML DBS service running on CMLs.

  • All ORS, CML as well as CRS, MPS and YPS services.

Follow chapter: Stop all ORS and CML services on all servers

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/warning.svg Outage

2

Make sure that count of data in all replication tables on all CMLs is equal to 0. Then stop YSoft SafeQ CML DBS services.

SELECT count(id) FROM cluster_sync_update_10;
SELECT count(id) FROM cluster_sync_update_20;
SELECT count(id) FROM cluster_sync_update_30;

images/s/en_GB/7901/58be3fa11e9ad58113c0ea45e7063389a7c7d344/_/images/icons/emoticons/warning.svg Outage

3

Update CML 2:

  • Use the time of CML update for next steps

Follow chapter: Update CML

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/information.svg The reason for starting with node 2 is to mitigate a risk of slow validation. The database on node 2 is quite small compared to CML 1 which makes it likely to validate faster. Time of validation defines duration of the outage.

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/warning.svg Outage

4

While step 2 is in progress, update one ORS in Near Roaming Group (NRG)

  • images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/information.svg Perform per each NRG

  • Stop services of updated and configured ORS

Follow chapter: Update ORS

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/warning.svg Outage

5

While step 2 is in progress, update standalone ORS servers

  • images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/information.svg Perform per each location

  • Stop services of updated and configured ORS

Follow chapter: Update ORS

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/warning.svg Outage

6

While step 2 is in progress, update ORS in every business critical location

  • Stop services of updated and configured ORS

Follow chapter: Update ORS

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/warning.svg Outage

7

After CML 2 is fully operational:

  • Start services on updated ORS servers from steps 3-5

  • Reinstall all Terminal Embedded devices that are connected to CML subsystem (not ORS)

 

None or partial

8

Verify functionality:

  • All updated ORS servers are connected to CML

  • Roaming print in each location works as expected (send/release job)

  • No exceptions in the logs

 

None or partial

MILESTONE 1

GO/NOGO DECISION

Decide whether proceed or roll back. Follow chapter "Rollback procedure for Milestone 1" for recovery procedure.

None or partial

9

Update CML 1 and all other CML nodes

Follow chapter: Update CML

None

10

Perform CML configuration changes in System Settings (if any)

Follow chapter: Additional steps

Intermittent service interruption

11

Update the rest of the ORS servers

Follow chapter: Update ORS

Intermittent service interruption

2. Before the installation

2.2. Expected system outage

Between steps 1 and 7, the printing is unavailable. It is expected to take up to ..... hours to restore it

2.3. Expected data loss

  • If executed as documented, no data loss is expected.

  • In case of rollback, data between step 1 (database backup) and rollback are lost. Users will need to reprint their jobs once the system is operational again.

3. Updating the environment

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/warning.svg The steps show manual changes. In real update procedure we suggest to have pre-configured files as suggested above.

3.1. Update CML

1) Backup the following as soon as services are stopped:

  • License.xml file from <CML>\conf folder

  • Both CML databases on both CML nodes

    • SQDB5

    • SQDB5_SQDW

2) Run the installer. 

Before running the installer make sure that the replication tables does not contain any data. In case some data are left in the tables they need to be either deleted or synchronized. Leaving the data in the tables during the update will have consequences on data flow between the nodes in the cluster and may eventually interrupt the service unexpectedly.

3) images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/warning.svg Wait for the validation to finish, do not interrupt! Click "retry" until <CML>\logs\validation-done.log is created with current date and time. It can take up to several hours if the database grew big.

4) Stop all services on updated CML 2 to perform all changes below (keep the database server service running, if it is local)

3.1.1. Set recovery for all Y Soft services

images/download/attachments/67046860/services-recovery.png

3.1.2. Start all CML Services

Start CML Service first; all other services in no particular order.

3.1.3. Activate the CML

Open Web browser and activate the system using offline activation (option on the right). Follow the instructions on the screen.

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/check.svg CML update is finished

3.1.4. Update YPS

Follow YPS specific documentation.

3.2. Update ORS

3.2.1. Backup

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/information.svg You can perform the ORS backups while CML is being installed

On each ORS run a backup script run.cmd

run.cmd

 

set SOURCE=C:\SafeQORSset DESTINATION=C:\SafeQORS-backupset EXCLUDE=C:\excl.txtxcopy %SOURCE%\* %DESTINATION%\* /s /q /y /EXCLUDE:%EXCLUDE%

excl.txt

 

logs\terminalserver\logs\terminalserver\AccountedJobsserver\temp\server\cache\server\spool\

3.2.2. Perform the update

  • (should be done in advance) Configure the safeq-ors.ini configuration file. The CML Server GUIDs can be found on the SafeQ Web Interface of the CML Servers 

  • Run the installORSv2-exe.cmd command file

  • Check the Install-ORS.log to make sure the installation has completed

  • Stop all ORS services

3.2.3. Set recovery for all services

Set this on all services on both CML as well as all ORS servers:

images/download/attachments/67046860/services-recovery2.png

 

3.2.4. Start ORS services

  • Start ORS Service

  • Start Terminal Server Service

  • Wait for Web Service to start by itself in case of NRG. Otherwise start manually

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/warning.svg Before starting other ORS:

  • Ensure there are no exceptions in logs of this one

  • Ensure they are connected using jConsole (or logs)

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/check.svg ORS update is finished

3.2.5. Update MPS

Follow MPS specific documentation.

3.3. Additional steps

At this point both CML servers are updated so configuration can be adjusted in System Settings.

3.3.1. Manually synchronize settings across CML servers

The new version of YSoft SafeQ might introduce new options that have their settings randomly generated during installation, e.g. security tokens. To prevent members of the cluster from having different values for these options, export and import settings on the CML1 (master server). This will then synchronize the same settings to all other CML servers. To do that:

  1. On the YSoft SafeQ admin web go to the System → System Settings section.

  2. Under the Actions menu in the top right corner of the page select Export settings to file.

  3. Save the generated file.

  4. Use the same Actions menu with the Import settings from file option.

  5. Choose the file from step 3 and click Import.

  6. Restart services on all other YSoft SafeQ servers (both CML and ORS).

3.3.2. Capture changes and update the Solution Reference Guide

  1. Collect <cml>\conf.jsp and update the Solution Reference Guide

  2. Export settings and update respective settings in Solution Reference Guide: 
    YSoft SafeQ Dashboard: System -> Actions -> Export settings to file

3.4. Update CRS

3.4.1. Requirements

Encryption enforced option on the SQL server is set to No

images/download/thumbnails/29011593/encryption.png

 

Somebody with the following permissions on their Windows account:

  • The following SQL Server Analysis Services roles are required: SQL Server Analysis Services "Server administrators" group

 

Y Soft service account has to be temporarily granted the following permissions. Permissions can be removed after the update is finished:

  • The following SQL Server Integration Services roles are required: db_ssisadmin

images/download/attachments/29011593/safeqdb_permissions.png

3.4.2. Backup

  • Run backup.bat for database dump

  • Replace <ENV>with respective environment name and run the command:
    backup.bat -env crs\<ENV>\node1

  • Stop the CRS service

  • Back the whole <CRS> directory up

3.4.3. Configure

In most of the cases it is possible to use the "old" DeploymentConfig package as the parameters do not change.

3.4.4. Run update script

Replace <ENV>with respective environment name and run the command:

update.bat -env crs\<ENV>\node1

This is the output of the installation:

images/download/attachments/29011593/2014-03-28_09_56_43-You_are_viewing__Merilou_De_Leon%27s_Desktop.png

Stop CRS service after the installation is done.

3.4.5. Post update steps

Start the CRS service

Send statistics from CML

images/download/attachments/29011593/send_stats.png

Wait for synchronization to finish (check this in CRS log)

Restart CRS service to force OLAP cubes processing

Check in crs.log if there are errors:

 

2014-03-28 08:04:35,641 INFO  mpleThreadPoolWorker-1|          CSDManagerCRS| Obtaining CDC Lock to generate statistics data...2014-03-28 08:04:35,643 INFO  mpleThreadPoolWorker-1|          CSDManagerCRS| CDC Lock obtained correctly2014-03-28 08:04:35,653 INFO  mpleThreadPoolWorker-1|          CSDManagerCRS| Running CSD optimization query #1 [csd_add_stats]2014-03-28 08:04:45,579 INFO  mpleThreadPoolWorker-1|          CSDManagerCRS| CSD Optimization query #1: 9921ms.2014-03-28 08:04:45,580 INFO  mpleThreadPoolWorker-1|          CSDManagerCRS| Running CSD optimization query #2 [build cube]2014-03-28 08:04:46,590 WARN  mpleThreadPoolWorker-1|      ConfigurationImpl| Requested property not available in cache. Key = 'cdc-proccess-limit'. Returning default: 'null'2014-03-28 08:11:00,668 DEBUG               Thread-2|          CSDManagerCRS| Microsoft (R) SQL Server Execute Package Utility2014-03-28 08:11:00,669 DEBUG               Thread-2|          CSDManagerCRS| Version 10.50.4270.0 for 64-bit2014-03-28 08:11:00,669 DEBUG               Thread-2|          CSDManagerCRS| Copyright (C) Microsoft Corporation 2010. All rights reserved.2014-03-28 08:11:00,670 DEBUG               Thread-2|          CSDManagerCRS|2014-03-28 08:11:00,670 DEBUG               Thread-2|          CSDManagerCRS| Started:  8:04:45 AM2014-03-28 08:11:00,670 INFO  mpleThreadPoolWorker-1|          CSDManagerCRS| Command succeeded [command="dtexec.exe" /SQL "\PackageBuildCubeSafeq" /MAXCONCURRENT " -1 " /CHECKPOINTING OFF /REPORTING E]2014-03-28 08:11:00,670 DEBUG               Thread-2|          CSDManagerCRS| DTExec: The package execution returned DTSER_SUCCESS (0).2014-03-28 08:11:00,670 INFO  mpleThreadPoolWorker-1|          CSDManagerCRS| CSD Optimization query #2: 375090ms.2014-03-28 08:11:00,670 DEBUG               Thread-2|          CSDManagerCRS| Started:  8:04:45 AM2014-03-28 08:11:00,671 DEBUG               Thread-2|          CSDManagerCRS| Finished: 8:11:00 AM2014-03-28 08:11:00,671 DEBUG               Thread-2|          CSDManagerCRS| Elapsed:  374.948 seconds2014-03-28 08:11:00,686 DEBUG mpleThreadPoolWorker-1|          CSDManagerCRS| CSD (MSSQL) optimization DONE in 385075

3.4.6. Troubleshooting

Connect to Analysis services -> open cube -> Data Sources -> Properties:

  • set safeqdb user for connection in "connection string" (test to confirm)

Changes in <CRS>\scripts\CUBE\4_\ in order to build the cube manually:

  • rename "full_cube_template_2008_std_sqlcred.xmla" to "full_cube_template.xmla" replacing the existing file

  • add pass and user, update server
    <ConnectionString>Provider=SQLOLEDB.1;Data Source=SQL-SWDS-SFQ-P1;Persist Security Info=True;Password=*******;User ID=safeqdb;Initial Catalog=yBoxDBCDC</ConnectionString>

If the installation fails, verify the required permissions and run the update again

images/download/attachments/29011593/process_terminated.png

 

4. Rollback procedure

4.1. Rollback procedure for Milestone 1

As a first step, collect data for analysis:

  • <CML/ORS>\*.log

  • <CML/ORS>\logs\*.log

  • <CML/ORS>\terminalserver\logs\*.log

  • Event viewer export from CML/ORS

  • CML 2 database backup

4.1.1. Recover printing availability where possible

  1. Stop all updated components

    1. All updated ORS servers

    2. Updated CML 2 server

  2. Start CML 1

  3. Start all not-yet-updated ORS servers

images/s/-3eliqb/8502/404359a7d2ab19c9c7c58d12013124a386b28257/_/images/icons/emoticons/thumbs_up.svg  At this point the system is partially operational

4.1.2. Restore ORS

  1. Restore ORS files from backup (do not un/reinstall)

  2. Start services:

    • Start ORS Service

    • Start Terminal Server Service

    • Wait for Web Service to start by itself

4.1.3. Restore CML

  1. Restore database on CML1 from backup performed

  2. Restore all binary files from backup created automatically during the installation

  3. Start CML Services

  4. Verify that the database is synchronizing

4.1.4. Restore CRS

  • Follow standard recovery procedures as documented

5 Updating cluster with network load balacing or Microsoft cluster services

Please get in touch with Customer support in order to get instructions specific for your environment.