S3 Classification Filtering

These procedures will test classification access controls for S3.

1. Requirements

Access to Keycloak admin console

2. Setup

2.1. Ensure Keycloak Group membership Mapper Enabled

In Keycloak, under the data-fabric realm,

Select the Client scopes menu.
Select profile scope.
Click the Mappers tab.
Click the Add mapper → By configuration button.
Select Group Membership.
Input:
- Name: groups
- Token Claim Name: groups
click Save.

If you get an error stating:

Could not create mapping: 'Protocol mapper exists with same name'

then the mapper is already configured.

2.2. Create Test Users

For the purposes of this test, we will create 3 users:

test-u - Test user with UNCLASSIFIED clearance, USA citizen.
test-s - Test user with SECRET clearance, USA citizen.
test-s-gbr - Test user with SECRET clearance, GBR citizen.

All data used in the test is synthetic and will be deleted at the end of the test to avoid confusion.

Login to the Keycloak admin console at https://DF_HOST/auth.
Navigate to realm data-fabric → Users.
Click the Add user button.
- Username: test-u
Click Create
Under the Attributes tab, click Add an attribute to enter each of the following key/value pairs:
- country: USA
- classification: UNCLASSIFIED
Click Save.
Under the Credentials tab, click Set password and input:
- Password: test
- Password confirmation: test
- Temporary: toggle to "Off"
Click Save → Save password (confirmation).

Repeat the steps above for user test-s, with the only difference being:

classification to SECRET.

Repeat the steps above for user test-s-gbr, with the only difference being:

country: GBR
classification: SECRET

3. Basic Tests

3.1. Authorization Token

Login to Data Fabric with the test-u / test username and password.
From the nav menu, click APIs to open the Swagger docs.
In the root Swagger listing under Fabric Services, click the /api/v1/s3 link.
Click the green Authorize button (right side). In the Available authorizations popup:
1. Logout of any/all currently authorized schemes.
2. Under the df_oauth authorization, type in df-minio for the client_id.
3. Click the Authorize button.
4. Verify the popup fills in for both client_id and client_secret, indicating a successful authorization flow.
5. Click any of Close to close the authorization popup.

3.2. List Buckets

In Swagger, expand the GET /buckets row and click the Try it out button.
Click the Execute button.

Verify you get back a 200 response with a list of buckets similar to:

[
  {
    "name": "df-notebooks",
    "creationDate": "2024-09-09T12:07:59.024Z"
  },
  {
    "name": "df-schemas",
    "creationDate": "2024-09-09T12:08:03.269Z"
  },
  {
    "name": "inbox-public",
    "creationDate": "2024-09-09T12:08:05.617Z"
  }
  ...
]

Bucket listing may vary by deployment environment.

3.3. Create a Bucket

In Swagger, expand the POST /buckets row and click the Try it out button.
In the Request body section, put
```
{
  "name": "test-bucket"
}
```
Click the Execute button.
Verify you get back a 200 response, indicating the test-bucket was created successfully.

3.4. Upload a File

In Swagger, expand the PUT /buckets/{bucketName}/objects/content row and click the Try it out button.
In the Parameters section, set:
- bucketName: test-bucket
- objectName: test-unmarked.txt
- Content-Type: text/plain
  
  Leave all other parameters unspecified.
In the Request body section, input:
```
This is an unmarked file.
```
Click the Execute button.

Verify you get back a 200 response with a JSON payload indicating your file was uploaded successfully.

{
 "name": "test-unmarked.txt",
 "etag": "06abb8b20a1a6323fbe9f8545fed35a5-1",
 "lastModified": "0001-01-01T00:00:00Z",
 "sizeBytes": 25,
 "expiration": "0001-01-01T00:00:00Z"
}

3.5. List Bucket Contents

In Swagger, expand the GET /buckets/{bucketName}/objects row and click the Try it out button.
In the Parameters section, set:
- bucketName: test-bucket
Click the Execute button.

Verify you get back a 200 response with a JSON payload listing the objects currently in the bucket.

[
 {
   "name": "test-unmarked.txt",
   "tags": {},
   "etag": "06abb8b20a1a6323fbe9f8545fed35a5-1",
   "lastModified": "2024-09-09T14:09:06.122Z",
   "sizeBytes": 25,
   "expiration": "0001-01-01T00:00:00Z"
 }
]

3.6. Download a File

In Swagger, expand the GET /buckets/{bucketName}/objects/content row and click the Try it out button.
In the Parameters section, set:
- bucketName: test-bucket
- objectName: test-unmarked.txt
Click the Execute button.
Verify you get back a 200 response with a "Download file" link under the Response body section.
Download and open the file, verifying it contains:
```
This is an unmarked file.
```

4. Classification Access Controls

In these steps, we will verify the S3 API supports classification control markings on a per-object basis.

In this scenario,

user test-u can mark and access objects up to U (unclassified).
user test-s can mark and access objects up to S (secret).
user test-s-gbr can mark and access objects up to S (secret), but only if it is also marked as releasable to his country (or FVEY).
all users can access any unmarked object.

4.1. Upload a File (`U`)

Logout/Login to DF as test-u and get an Authorization Token.
In Swagger, expand the PUT /buckets/{bucketName}/objects/content row and click the Try it out button.
In the Parameters section, set:
- bucketName: test-bucket
- objectName: test-unclassified.txt
- Content-Type: text/plain
- classification: U
  
  Leave all other parameters unspecified.
In the Request body section, input:
```
This is an unclassified file.
```
Click the Execute button.

Verify you get back a 200 response with a JSON payload indicating your file was uploaded successfully.

{
 "name": "test-unclassified.txt",
 "etag": "c46c2451fed80da5bf775c0b78ee1ac1-1",
 "lastModified": "0001-01-01T00:00:00Z",
 "sizeBytes": 29,
 "expiration": "0001-01-01T00:00:00Z"
}

4.2. Upload a File (`S//NF`)

Logout/Login to DF as test-s and get an Authorization Token.
In Swagger, expand the PUT /buckets/{bucketName}/objects/content row and click the Try it out button.
In the Parameters section, set:
- bucketName: test-bucket
- objectName: test-secret-nf.txt
- Content-Type: text/plain
- classification: S//NF
  _Leave all other parameters unspecified._

In the Request body section, input:

This is a secret file with NOFORN release.

Click the Execute button.

Verify you get back a 200 response with a JSON payload indicating your file was uploaded successfully.

{
 "name": "test-secret-nf.txt",
 "etag": "fa9e9e9e2f26ba627a4916495df443ff-1",
 "lastModified": "0001-01-01T00:00:00Z",
 "sizeBytes": 42,
 "expiration": "0001-01-01T00:00:00Z"
}

4.3. Upload a File (`S//REL TO USA, FVEY`)

Logout/Login to DF as test-s and get an Authorization Token.
In Swagger, expand the PUT /buckets/{bucketName}/objects/content row and click the Try it out button.
In the Parameters section, set:
- bucketName: test-bucket
- objectName: test-secret-fvey.txt
- Content-Type: text/plain
- classification: S//NF
  _Leave all other parameters unspecified._

In the Request body section, input:

This is a secret file releasable to FVEY.

Click the Execute button.
Verify you get back a 200 response with a JSON payload indicating your file was uploaded successfully.

{
 "name": "test-secret-fvey.txt",
 "etag": "8ae7c40426847b14f4d54548cc5b1f6f-1",
 "lastModified": "0001-01-01T00:00:00Z",
 "sizeBytes": 41,
 "expiration": "0001-01-01T00:00:00Z"
}

4.4. List Bucket Contents

In Swagger, expand the GET /buckets/{bucketName}/objects row and click the Try it out button.
In the Parameters section, set:
- bucketName: test-bucket
Click the Execute button.

Verify you get back a 200 response with a JSON payload listing the objects currently in the bucket, with their normalized classification markings.

[
  {
    "name": "test-secret-fvey.txt",
    "tags": {},
    "etag": "8ae7c40426847b14f4d54548cc5b1f6f-1",
    "classification": {
      "raw": "S//REL TO USA, FVEY",
      "components": {
        "classification": "S",
        "disseminationControls": [
          "REL"
        ],
        "ownerProducer": [
          "USA"
        ],
        "releasableTo": [
          "USA",
          "FVEY"
        ]
      }
    },
    "lastModified": "2024-09-09T18:12:28.222Z",
    "sizeBytes": 41,
    "expiration": "0001-01-01T00:00:00Z"
  },
  {
    "name": "test-secret-nf.txt",
    "tags": {},
    "etag": "0e867d585a862b6c0d96bb767a90cfa3-1",
    "classification": {
      "raw": "S//NF",
      "components": {
        "classification": "S",
        "disseminationControls": [
          "NF"
        ],
        "ownerProducer": [
          "USA"
        ],
        "releasableTo": []
      }
    },
    "lastModified": "2024-09-09T18:11:48.511Z",
    "sizeBytes": 42,
    "expiration": "0001-01-01T00:00:00Z"
  },
  {
    "name": "test-unclassified.txt",
    "tags": {},
    "etag": "c46c2451fed80da5bf775c0b78ee1ac1-1",
    "classification": {
      "raw": "U",
      "components": {
        "classification": "U",
        "disseminationControls": [],
        "ownerProducer": [
          "USA"
        ],
        "releasableTo": []
      }
    },
    "lastModified": "2024-09-09T18:09:23.965Z",
    "sizeBytes": 29,
    "expiration": "0001-01-01T00:00:00Z"
  },
  {
    "name": "test-unmarked.txt",
    "tags": {},
    "etag": "06abb8b20a1a6323fbe9f8545fed35a5-1",
    "lastModified": "2024-09-09T18:07:43.495Z",
    "sizeBytes": 25,
    "expiration": "0001-01-01T00:00:00Z"
  }
]

4.5. Verify `test-u` Access Controls

This verifies a user of a lower classification cannot read content marked with a higher classification.

Logout/Login to DF as test-u and get an Authorization Token.
In Swagger, expand the GET /buckets/{bucketName}/objects/content row and click the Try it out button.
In the Parameters section, set:
- bucketName: test-bucket
- objectName: test-unclassified.txt
Click the Execute button.
Verify you get back a successful 200 indicating the user can access the unclassified file.
In the Parameters section, change:
- objectName: test-secret-nf.txt
Click the Execute button.
Verify you get back a 403 stating
```
Access Denied
```
indicating the user cannot access the file due to lack of clearance.
In the Parameters section, change:
- objectName: test-secret-fvey.txt
Click the Execute button.
Verify you get back a 403 stating
```
Access Denied
```

indicating the user cannot access the file due to lack of clearance.

4.6. Verify `test-s` Access Controls

This verifies a user can read marked content only if they are cleared for it.

Logout/Login to DF as test-s and get an Authorization Token.
In Swagger, expand the GET /buckets/{bucketName}/objects/content row and click the Try it out button.
In the Parameters section, change:
- bucketName: test-bucket
- objectName: test-secret-nf.txt
Click the Execute button.
Verify you get back a successful 200 indicating the user can access the S//NF file.
In the Parameters section, change:
- objectName: test-secret-fvey.txt
Click the Execute button.
Verify you get back a successful 200 indicating the user can access the S//REL TO USA, FVEY file.

4.7. Verify `test-s-gbr` Access Controls

Logout/Login to DF as test-s-gbr and get an Authorization Token.
In Swagger, expand the GET /buckets/{bucketName}/objects/content row and click the Try it out button.
In the Parameters section, change:
- bucketName: test-bucket
- objectName: test-secret-nf.txt
Click the Execute button.
Verify you get back a 403 stating
```
Access Denied
```

indicating the user cannot access the file since he is a foreign citizen and the file is marked NOFORN. . In the Parameters section, change: * objectName: test-secret-fvey.txt . Click the Execute button. . Verify you get back a successful 200 indicating the user can access the file since it is releasable to FVEY (which he is a citizen of a member country).

5. Cleanup

6. Troubleshooting

6.1. Keycloak `JWTGroupsParam` Error

If you get a response saying,

Failed to get JWTGroupsParam. jwtGroupsStr: [] Error: <nil>. You may need to setup a group membership mapper in Keycloak. Go to Keycloak -> Data Fabric Realm -> Client Scopes -> profile -> Mappers -> Add Mapper -> By Configuration -> Group Membership -> Name = "groups", Token Claim Name = "groups"

then just do what it says.

6.2. Expired Token Error

If you get an unexpected 401 response, and any of the following are true:

the Response headers include:

www-authenticate: Bearer error="invalid_token",error_description="Jwt expired at 2024-09-09T13:50:04Z",error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"

-OR- the Response body states:
```
Failed to validate user authorization token with OIDC provider.
```
then you need a new authorization token. Repeat the Authorization Token steps.

S3 Classification Filtering

1. Requirements

2. Setup

2.1. Ensure Keycloak Group membership Mapper Enabled

2.2. Create Test Users

3. Basic Tests

3.1. Authorization Token

3.2. List Buckets

3.3. Create a Bucket

3.4. Upload a File

3.5. List Bucket Contents

3.6. Download a File

4. Classification Access Controls

4.1. Upload a File (U)

4.2. Upload a File (S//NF)

4.3. Upload a File (S//REL TO USA, FVEY)

4.4. List Bucket Contents

4.5. Verify test-u Access Controls

4.6. Verify test-s Access Controls

4.7. Verify test-s-gbr Access Controls

5. Cleanup

6. Troubleshooting

6.1. Keycloak JWTGroupsParam Error

6.2. Expired Token Error

4.1. Upload a File (`U`)

4.2. Upload a File (`S//NF`)

4.3. Upload a File (`S//REL TO USA, FVEY`)

4.5. Verify `test-u` Access Controls

4.6. Verify `test-s` Access Controls

4.7. Verify `test-s-gbr` Access Controls

6.1. Keycloak `JWTGroupsParam` Error