Skip to content

Replication

Danger

This is a beta feature. Production use is highly discouraged.

The component Sync provides the ability to replicate data between Cloudrexx websites.

Architecture

The Sync component listens to model changes and pushes them to any configured remote (another Cloudrexx installation having Sync set up) through the RESTful-API (DataAccess). For proper foreign key handling, the Sync component injects a wrapper command (called sync) on top of the RESTful-API command v1. Meaning, instead of pushing data directly to /api/v1/<outputModule>/<dataAccessName>/<elementId> on the remote, data is pushed to /api/sync/v1/<outputModule>/<dataAccessName>/<elementId> (note the endpoint /sync) instead.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
╔════════════════════════════╗
║ Cloudrexx A                ║
║                            ║
║        ┌──────────┐        ║
║        │ Database │        ║
║        └──────────┘        ║
║             │              ║
║   ┌────────────────────┐   ║
║   │ DoctrineRepository │   ║
║   └────────────────────┘   ║
║             │              ║
║       ┌────────────┐       ║
║       │ DataSource │       ║
║       └────────────┘       ║
║             │              ║
║       ┌────────────┐       ║
║       │ DataAccess │       ║
║       └────────────┘       ║
║             │              ║
║         ┌──────┐           ║
║         │ Sync │           ║
║         └──────┘           ║
║             │              ║
╚════════════════════════════╝
              │
              ⇅ HTTP
              │
╔════════════════════════════╗
║             │              ║
║         ┌──────┐           ║
║         │ Sync │           ║
║         └──────┘           ║
║             │              ║
║       ┌────────────┐       ║
║       │ DataAccess │       ║
║       └────────────┘       ║
║             │              ║
║       ┌────────────┐       ║
║       │ DataSource │       ║
║       └────────────┘       ║
║             │              ║
║   ┌────────────────────┐   ║
║   │ DoctrineRepository │   ║
║   └────────────────────┘   ║
║             │              ║
║        ┌──────────┐        ║
║        │ Database │        ║
║        └──────────┘        ║
║                            ║
║ Cloudrexx B                ║
╚════════════════════════════╝

Supported DataSource

The Sync component does currently only support the replication of the DataSource DoctrineRepository.

Setup

To enable the replication of model data two steps have to be taken:

  1. Enable RESTful-API on Replica
  2. Enable Sync on Primary

Enable RESTful-API on Replica

On the Cloudrexx installation that will act as the replica, do as follows:

  1. Create an API endpoint for the model that shall be replicated:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
    INSERT INTO `contrexx_core_data_source`
    SET `identifier` = 'MODEL',
        `options` = 'a:0:{}',
        `type` = 'doctrineRepository';
INSERT INTO `contrexx_core_module_data_access`
    SET `data_source_id` = (
            SELECT `id` FROM `contrexx_core_data_source` WHERE `identifier` = 'MODEL'
        ),
        `name` = 'ENDPOINT',
        `field_list` = 'a:0:{}',
        `access_condition` = 'a:0:{}',
        `allowed_output_methods` = 'a:0:{}';

    Note

    Replace MODEL by the FQCN of the model to replicate. E.g.: Cx\\Core\\Routing\\Model\\Entity\\RewriteRule
    Replace ENDPOINT by a unique name to identify the endpoint. E.g.: RewriteRule

  2. Create a new API-Key (under Administration > RESTful API) and assign the newly created endpoint (e.g. RewriteRule) having write access

Enable Sync on Primary

On the Cloudrexx installation that will act as the primary, do as follows:

  1. Create the same API endpoint as the one created on the replica:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
    INSERT INTO `contrexx_core_data_source`
    SET `identifier` = 'MODEL',
        `options` = 'a:0:{}',
        `type` = 'doctrineRepository';
INSERT INTO `contrexx_core_module_data_access`
    SET `data_source_id` = (
            SELECT `id` FROM `contrexx_core_data_source` WHERE `identifier` = 'MODEL'
        ),
        `name` = 'ENDPOINT',
        `field_list` = 'a:0:{}',
        `access_condition` = 'a:0:{}',
        `allowed_output_methods` = 'a:0:{}';

    Note

    Replace MODEL and ENDPOINT by the same values used on the replica.

  2. Enable replication of the newly created endpoint:

    1
2
3
4
5
6
7
8
    INSERT INTO `contrexx_core_module_sync`
SET
    `data_access_id` = (
        SELECT `id` FROM `contrexx_core_data_source` WHERE `identifier` = 'MODEL'
    ),
    `to_uri` = '',
    `api_key` = '',
    `active` = '1';

    Note

    Replace MODEL by the same value used to create the endpoint.

  3. Register the replica:

    1
2
3
4
5
6
7
    INSERT INTO `contrexx_core_module_sync_host`
    SET `host` = 'https://HOST',
        `active` = 1,
        `api_key` = 'API_KEY',
        `api_version` = 1,
        `url_template` = '',
        `state` = 0;

    Note

    Replace HOST by the domain name of the replica (for local environments using cx env see Connect multiple ENVs to each other).
    Replace API_KEY by the previously generated API-Key on the replica.

  4. Enable replication of the newly created endpoint to the replica:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
    INSERT INTO `contrexx_core_module_sync_host_entity`
SET `sync_id` = (
        SELECT `id` FROM `contrexx_core_module_sync`
        WHERE `data_access_id` = (
            SELECT `id` FROM `contrexx_core_data_source` WHERE `identifier` = 'MODEL'
        ) AND `active` = '1'
    ),
    `host_id` = (
         SELECT `id` FROM `contrexx_core_module_sync_host` WHERE `host` = 'https://HOST'
    ),
    `entity_id` = '*';

    Note

    Replace MODEL by the same value used to create the endpoint.
    Replace HOST by the name of the replica (must be identical to the one used in the previous step).

Special Case: Calendar

As the model of the Calendar component is not purely a DoctrineRepository DataSource (Remember: DoctrineRepository is the only supported DataSource), but instead a mix of LegacyDatabaseRepository and DoctrineRepository, the setup is a bit different.

  1. On the replica do as follows:

    1. Add a new API-Key and assign all calendar-* endpoints (with write access)

  2. On the primary do as follows:

    1. Enable endpoint (calendar-event) for replication:
    1
2
3
4
5
    UPDATE `contrexx_core_module_sync`
    SET `active` = '1'
    WHERE `data_access_id` = (
        SELECT `id` FROM `contrexx_core_module_data_access` WHERE `name` = 'calendar-event'
    );
    1. Enable publishing function in Calendar component:
    1
2
3
    UPDATE `contrexx_module_calendar_settings`
    SET `value` = '1'
    WHERE `name` = 'publicationStatus';
    1. Register the replica:
    1
2
3
4
5
6
7
    INSERT INTO `contrexx_core_module_sync_host`
    SET `host` = 'https://HOST',
        `active` = 1,
        `api_key` = 'API_KEY',
        `api_version` = 1,
        `url_template` = '',
        `state` = 0;

    Note

    Replace HOST by the domain name of the replica (for local environments using cx env see Connect multiple ENVs to each other).
    Replace API_KEY by the previously generated API-Key on the replica.

    1. Register replica as publishing-target in Calendar component:
    1
2
3
4
5
6
7
    INSERT INTO `contrexx_module_calendar_host`
    SET `title` = 'Replica',
        `uri` = 'https://HOST',
        `cat_id` = 0,
        `key` = '',
        `confirmed` = 0,
        `status` = 0);

    Note

    Replace HOST by the name of the replica (must be identical to the one used in the previous step).

Bidirectional Replication

Note

For easier understanding the terms primary and replica refer to the Cloudrexx installations acting as primary and replica of the initial instructions from above. Meaning that when setting up the bidirectional replication (according to the following instructions) the term primary technically refers to the installation acting as the replica and replica to the installation acting a primary.

To enable bidirectional replication, simply set the replication up the other way round. Meaning:

  1. Create a new API-Key (but this time on primary) and assign the previously created endpoint (e.g. RewriteRule) having write access
  2. Enable synchronization of the endpoint on the replica
  3. Register the primary on the replica as replication target
  4. On the replica, enable replication of the endpoint to primary

Important

Bidirectional replication only works properly, if the HOST on both sides are set to the main domain (configuration option Main domain under Administration > Global Configuration > System) of the other side.

Deregistration

Deregistration of a replica involves the following steps:

  1. Flush data from replica
  2. Disable RESTful-API on replica
  3. Deregister replica from primary

Flush data from replica

Note

This step can be skipped, in case the replicated data should be kept on the replica after deregistration.

To remove any replicated data from the replica do:

TODO

Document process to remove replicated data from replica

Disable RESTful-API on replica

TODO

Document process to disable RESTful-API on replica

Deregister replica from primary

Tip

See Special Case: Calendar

To completely purge a replica from a primary do:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
DELETE
    `sync_host`,
    `sync_entity`,
    `change`
FROM
    `contrexx_core_module_sync_host` AS `sync_host`
INNER JOIN `contrexx_core_module_sync_host_entity` AS `sync_entity`
ON
    `sync_entity`.`host_id` = `sync_host`.`id`
INNER JOIN `contrexx_core_module_sync_change_host` AS `change`
ON
    `change`.`host_id` = `sync_host`.`id`
WHERE
    `sync_host`.`host` = 'https://HOST';

Note

Replace HOST by the name of the replica

Special Case: Calendar

Remove replica from primary:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
DELETE
    `sync_host`,
    `change`,
    `calendar_host`,
    `event`
FROM
    `contrexx_core_module_sync_host` AS `sync_host`
LEFT OUTER JOIN `contrexx_core_module_sync_change_host` AS `change`
ON
    `change`.`host_id` = `sync_host`.`id`
LEFT OUTER JOIN `contrexx_module_calendar_host` AS `calendar_host`
ON
    `calendar_host`.`uri` = `sync_host`.`host`
LEFT OUTER JOIN `contrexx_module_calendar_rel_event_host` AS `event`
ON
    `event`.`host_id` = `calendar_host`.`id`
WHERE
    `sync_host`.`host` = 'https://HOST';

Note

Replace HOST by the name of the replica