OpenStack Keystone Fernet Key in Multiple Regions

Introduce Keystone Fernet Token (2)

Based on my previous blog:

http://gogosatellite.blogspot.tw/2016/05/keystone-setup-fernet-token-and.html

One can using key-rotate tool: keystone-manage fernet_rotate

root@controller:~# keystone-manage fernet_rotate --keystone-user keystone --keystone-group keystone
2016-05-27 08:29:01.388 2400 WARNING keystone.token.providers.fernet.utils [-] [fernet_tokens] key_repository is world readable: /etc/keystone/fernet-keys/
2016-05-27 08:29:01.404 2400 INFO keystone.token.providers.fernet.utils [-] Starting key rotation with 2 key files: ['/etc/keystone/fernet-keys/0', '/etc/keystone/fernet-keys/1']
2016-05-27 08:29:01.404 2400 INFO keystone.token.providers.fernet.utils [-] Current primary key is: 1
2016-05-27 08:29:01.405 2400 INFO keystone.token.providers.fernet.utils [-] Next primary key will be: 2
2016-05-27 08:29:01.406 2400 INFO keystone.token.providers.fernet.utils [-] Promoted key 0 to be the primary: 2
2016-05-27 08:29:01.408 2400 INFO keystone.token.providers.fernet.utils [-] Created a new key: /etc/keystone/fernet-keys/0

We will explane that how is the key rotation later

root@controller:~# ls /etc/keystone/fernet-keys/
0  1  2

root@controller:~# cat /etc/keystone/fernet-keys/2
BIoWWHPzDcoNRhFsg3TFzrRHoYVlL1MECDreHBREqJo=

root@controller:~# cat /etc/keystone/fernet-keys/0
RuiOVQGerT_owvluKjEaN9mOzy52vTJLCujAyfYkCrQ=

Even using the same key, the token is still different, since the change is due to expired time that is related to created time. So the token is always different.

The following shows the new token after rotating the fernet keys.

junmeindeMacBook-Pro:openstack_api junmein$ bash getToken.sh
HTTP/1.1 201 Created
Date: Fri, 27 May 2016 01:02:20 GMT
Server: Apache/2.4.7 (Ubuntu)
X-Subject-Token: gAAAAABXR5yfF-PkUJPVRFA2gCDsZS6YT5y9Ud4NeTd7RLRaxFq8dp_om3jdCdJI2-ncQ5CFDHqZDRhAaeIS3BlfXoqkE9CUBX4jdUf4k6WwwCm4nkH52kpAEAkJuXntfs9qxweWPESaGJ-_bFJGWsuk9MniJDMCWXulml4MLcNG8zsHc98cVsg%3D
Vary: X-Auth-Token
X-Distribution: Ubuntu
x-openstack-request-id: req-b942080e-b242-4199-bf79-31d8d3111900
Content-Length: 2065
Content-Type: application/json

{"token": {"methods": ["password"], "roles": [{"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}], "expires_at": "2016-05-27T02:02:22.229081Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "258b879e4df748caa1bac3416d38a819", "name": "testtenant"}, "catalog": [{"endpoints": [{"region_id": "RegionOne", "url": "http://controller:35357/v2.0", "region": "RegionOne", "interface": "admin", "id": "02f744638b2f44edb28bd64d9894c34a"}, {"region_id": "RegionOne", "url": "http://controller:5000/v2.0", "region": "RegionOne", "interface": "public", "id": "34171612079f4b028fdb01e38ec04b8c"}, {"region_id": "RegionTwo", "url": "http://controller1:5000/v2.0", "region": "RegionTwo", "interface": "internal", "id": "5222cc1e398043ffae73f74caa66a451"}, {"region_id": "RegionTwo", "url": "http://controller1:5000/v2.0", "region": "RegionTwo", "interface": "public", "id": "55b13c6057c2401fbff156b1b3d81db1"}, {"region_id": "RegionTwo", "url": "http://controller1:35357/v2.0", "region": "RegionTwo", "interface": "admin", "id": "d68629a1967b4c81aa3717b0d4931458"}, {"region_id": "RegionOne", "url": "http://controller:5000/v2.0", "region": "RegionOne", "interface": "internal", "id": "dacd582311a641b9ade00421ab19e8a2"}], "type": "identity", "id": "97fb94ea818d452894d33b900781b98e", "name": "keystone"}, {"endpoints": [{"region_id": "RegionTwo", "url": "http://lala/v2", "region": "RegionTwo", "interface": "admin", "id": "34188152080646b99f974ed7b05b6a58"}, {"region_id": "RegionTwo", "url": "http://sasa/v2", "region": "RegionTwo", "interface": "internal", "id": "f02b47a39d7449c693982b837f739537"}, {"region_id": "RegionTwo", "url": "http://haha/v2", "region": "RegionTwo", "interface": "public", "id": "f3e80838241541138e6bc31b6528a7d2"}], "type": "service1", "id": "8f1ce2e503ba4fbcb095e8469200b8e4", "name": "service1"}], "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "d8b0e3b8824143b0b47b8155747fa560", "name": "newuser"}, "audit_ids": ["Bbkxxh9PTGqQzCGiOsf4iA"], "issued_at": "2016-05-27T01:02:23.000000Z"}}

export token and get tenant

junmein$ export TOKEN=gAAAAABXR5yfF-PkUJPVRFA2gCDsZS6YT5y9Ud4NeTd7RLRaxFq8dp_om3jdCdJI2-ncQ5CFDHqZDRhAaeIS3BlfXoqkE9CUBX4jdUf4k6WwwCm4nkH52kpAEAkJuXntfs9qxweWPESaGJ-_bFJGWsuk9MniJDMCWXulml4MLcNG8zsHc98cVsg%3D

junmein$ cat getTenant.sh
curl -H "X-Auth-Token: $TOKEN" http://172.16.235.128:5000/v2.0/tenants

To check Token validate or not by geting tenant information.

junmein$ bash getTenant.sh
{"tenants_links": [], "tenants": [{"description": "atesttenant", "enabled": true, "id": "258b879e4df748caa1bac3416d38a819", "name": "testtenant"}]}

Key Rotation Method

Thanks for this blog, I just spend 5 mins I then can understand the key rotation method.

http://lbragstad.com/fernet-tokens-and-key-distribution/

We explan how the Key repository evolved from initial to roration.

We have 3 states

• The Highest index (>=1) is primary (ready to become secondary, used for encryption and decryption)
• The lower index (>0) is secondary (ready to purged, decryption only)
• The lowest index(=0) is stage (ready to become primary, decryption only)

The evolution shown in following table.

Action	Key State
initial	0 1
rotate	0 (new) 1 (old) 2 (from 0)
rotate1	0(new) 2(old) 3 (from 0) ;1(purged)
rotate2	0(new) 3(old) 4(from 0); 2(purged)

• As we know stage 0 will become primary after rotate, so we note the primary with (from 0) to identify the highest index is from stage 0.
• The number will be increased.
• After becoming primary, the stage will be generated a new index 0 and with a new key.
• If setting max key=3, after several rotation the lower index will be purged during the evolution.

Multiple Region testing using Fernet Token

In Region1: 172.16.235.128
In Region2: 172.16.235.159

Exp. 1:

Assume: We set the directory /etc/keystone/fernet-keys the same in both of region1 and region2. Assume: Keystone DB are the same for all the user information.

junmein$ cat getTenant2.sh
curl -H "X-Auth-Token: $TOKEN" http://172.16.235.159:5000/v2.0/tenants

junmein$ bash getTenant2.sh
{"tenants_links": [], "tenants": [{"description": "atesttenant", "enabled": true, "id": "258b879e4df748caa1bac3416d38a819", "name": "testtenant"}]}junmeindeMacBook-Pro:openstack_api

junmein$ bash getTenant.sh
{"tenants_links": [], "tenants": [{"description": "atesttenant", "enabled": true, "id": "258b879e4df748caa1bac3416d38a819", "name": "testtenant"}]}junmeindeMacBook-Pro:openstack_api

Now we can access two regions using one token.

Exp. 2:

Assume: Keystone DB are the same for all the user information. action: fernet key rotation in region 1

root@controller:~# keystone-manage fernet_rotate --keystone-user keystone --keystone-group keystone
2016-05-27 09:55:09.757 1791 WARNING keystone.token.providers.fernet.utils [-] [fernet_tokens] key_repository is world readable: /etc/keystone/fernet-keys/
2016-05-27 09:55:09.759 1791 INFO keystone.token.providers.fernet.utils [-] Starting key rotation with 3 key files: ['/etc/keystone/fernet-keys/0', '/etc/keystone/fernet-keys/1', '/etc/keystone/fernet-keys/2']
2016-05-27 09:55:09.759 1791 INFO keystone.token.providers.fernet.utils [-] Current primary key is: 2
2016-05-27 09:55:09.759 1791 INFO keystone.token.providers.fernet.utils [-] Next primary key will be: 3
2016-05-27 09:55:09.760 1791 INFO keystone.token.providers.fernet.utils [-] Promoted key 0 to be the primary: 3
2016-05-27 09:55:09.761 1791 INFO keystone.token.providers.fernet.utils [-] Created a new key: /etc/keystone/fernet-keys/0
2016-05-27 09:55:09.762 1791 INFO keystone.token.providers.fernet.utils [-] Excess keys to purge: [1]

Now the key repostory of region1 and region 2 is different.

Again we use the same Token to access region1 and region2.

Accessing Region1

junmein$ bash getTenant.sh
{"tenants_links": [], "tenants": [{"description": "atesttenant", "enabled": true, "id": "258b879e4df748caa1bac3416d38a819", "name": "testtenant"}]}junmeindeMacBook-Pro:openstack_api

Accessing Region2

junmein$ bash getTenant2.sh
{"tenants_links": [], "tenants": [{"description": "atesttenant", "enabled": true, "id": "258b879e4df748caa1bac3416d38a819", "name": "testtenant"}]}junmeindeMacBook-Pro:openstack_api

That is satify the fernet key rotation algorithm that both of region can be access even using the token generated before key rotation.

Exp. 3

Assume: Keystone DB are the same for all the user information. action: fernet key rotation in region 1 again.

keystone-manage fernet_rotate --keystone-user keystone --keystone-group keystone
2016-05-27 09:59:43.946 1806 WARNING keystone.token.providers.fernet.utils [-] [fernet_tokens] key_repository is world readable: /etc/keystone/fernet-keys/
2016-05-27 09:59:43.948 1806 INFO keystone.token.providers.fernet.utils [-] Starting key rotation with 3 key files: ['/etc/keystone/fernet-keys/0', '/etc/keystone/fernet-keys/2', '/etc/keystone/fernet-keys/3']
2016-05-27 09:59:43.949 1806 INFO keystone.token.providers.fernet.utils [-] Current primary key is: 3
2016-05-27 09:59:43.949 1806 INFO keystone.token.providers.fernet.utils [-] Next primary key will be: 4
2016-05-27 09:59:43.950 1806 INFO keystone.token.providers.fernet.utils [-] Promoted key 0 to be the primary: 4
2016-05-27 09:59:43.951 1806 INFO keystone.token.providers.fernet.utils [-] Created a new key: /etc/keystone/fernet-keys/0
2016-05-27 09:59:43.952 1806 INFO keystone.token.providers.fernet.utils [-] Excess keys to purge: [2]

In Region1's key repository

root@controller:~# ls /etc/keystone/fernet-keys/
0  3  4

Our Token is generated by using index 1. Now 1 has been purged after key rotation. So we cannot access to Region1. The Token has been expired.

bash getTenant.sh
{"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}

But in Region2 index 1 key is still in the key repository, so we can connect to Region.

root@controller2:~# ls /etc/keystone/fernet-keys/
0  1  2

bash getTenant2.sh
{"tenants_links": [], "tenants": [{"description": "atesttenant", "enabled": true, "id": "258b879e4df748caa1bac3416d38a819", "name": "testtenant"}]}junmeindeMacBook-Pro:openstack_api

Conclusion for Multi-Rgions with Keystone Fernet Key

1. Syn user data by DB replication is necessary, since we need to keep all the user information are the same.
2. We don't need to sync token, very heavy, that will help the system more stable.
3. Using same token to multiple regions is validate with both same token, and same key repository.
4. After rotate, assume region 1, two region can still works using the same token, and new token. Since encryption key still in the keys(for decryption) of repository for both region.
5. Sync key repository are necessary, but not instantly, we can do it before rotate twice.
6. After rotate twice, if region2 lost sync twice, some key will be purged and some token will be expired. And some token will be invalidate in region2 or region1, since encryption key is not existed in both key repository(for decrption).

The Process is to multiple region is

1. Replicate keystone DB, including all users information.
2. sync initial key repository, to make sure both regions are the same.
3. rotate
4. sync key repository (No need to instantly).