Encryption

[odbuser2]

Will encryption be supported at various levels in ArcadeDB?

e.g.
By database for encryption at rest
By type
By property to protect specific data

[odbuser2]

Any thoughts on this? In particular support for bcfips via the standard java.security.Provider could widen the use of arcadedb.

[odbuser2]

I should have qualified the request with the need to have encryption at rest (db level). Not encryption to satisfy every possible case (e.g. field level, rest, etc.).

Application handling of encryption is ok for "field" level encrypted but it will not address encryption at rest of the entire database. Orientdb supported this between 2.2 and 3 (I thought it was also in 3+ but maybe not?).

This shouldn't be hard as it could be a wrapper around the file io. @lvca maybe you could point out the locations and any gotchas? I can take a look.

Without "at rest" encryption via java.security, arcade can't be used in a huge chunk of the industry.

[lvca]

@odbuser2 We could support encryption at rest at the page level, so it would be efficient. The issue is with the size: the page has a fixed size, but with encryption it becomes variable. If it's shorter, we could fill it with blanks, but if larger we can't store it with the current architecture.

[lvca]

Silly question: what about using an encrypted folder to store the ArcadeDB database? In this way, the OS file system takes care of the encryption and it would be transparent for ArcadeDB.

[odbuser2]

Not a silly question. There are a few issues with using encrypted folders:

1. Each OS supports a different means to support encrypted folders
2. The OS configuration for the encryption is usually minimal or non-existent
3. The OS does not keep with standard best practices (at least until each major update)
4. 3rd party support for encrypted folders have similar issues (unless we write it ourselves)
5. Encrypted folders are susceptible to man-in-the middle exposure (e.g. layered file systems where a layer before the encryption layer is compromised)
6. Key and salt retrieval is not customizable and often does not meet standards

But what could be done (and probably should be done on pretty much all java apps) is to use a pluggable FileSystem for all arcadedb operations (can't simply override the default one since there would be wide implications for that). That way, any number of filesystems could be implemented (e.g. in memory, encrypted, etc.).

RandomAccessFile would also need to be pluggable and there may be limitations imposed.

Would you be willing to make the FileSystem and RandomAccessFile pluggable?

[lvca]

@odbuser2 Having a pluggable file system is certainly possible, there are only a few components that go IO with the file system. On the other side, the easiest thing would be compressing the records before they are written into the page and uncompressing them just after they are read from the file system. @odbuser2 if that is the case, where would you store the encryption credentials?

[odbuser2]

if that is the case, where would you store the encryption credentials?

That's a secret ;0).

Are you suggesting to encrypt while writing to the page? I guess that would work. If it's pluggable, the secret(s) can be stored in a number of places but I'd suggest not even addressing that. Making a pluggable api for writing to a page made be enough and the implementor can control the entire encryption process. But I reserve judgement until I understand what it means to write to the page.

[lvca]

ArcadeDB already supports custom event listeners for Create, Update, and Delete operations. It was missing the Read to provide a complete API to manipulate the record on read. Now it's here: #1175.

Example of registering an event (taken from our test suite

arcadedb/engine/src/test/java/com/arcadedb/event/DatabaseEventsTest.java

Line 74 in 3969df8

public void testAfterCreate() {

):

@Test
  public void testAfterCreate() {
    final AtomicInteger counter = new AtomicInteger();
    final AfterRecordCreateListener listener = record -> counter.incrementAndGet();

    database.getEvents().registerListener(listener);
    try {
      // CREATE SOME RECORDS HERE
    } finally {
      database.getEvents().unregisterListener(listener);
    }
  }

Events can be listened to at the database or type level.

@odbuser2 you could hook up here to provide your own encryption.

[odbuser2]

This is not encryption at rest. This is field level encryption. The schema is not encrypted. A pluggable file system and pluggable RandomFileAccess should allow an implementation of encryption at rest.

[lvca]

You're right, dunno why I said at rest. I renamed the test to RecordEncryptionTest.

[lvca]

With the last changes, you can implement your own Encryption at rest. Here is a full example:

arcadedb/engine/src/test/java/com/arcadedb/event/EncryptionAtRestTest.java

Line 51 in afc2834

private final static String ALGORITHM = "AES/CBC/PKCS5Padding";

[dijef]

I tried implementing encryption using events as in example you provided but it is not possible to make it work with transactions.
It's fine when reading vertex but not when writing to it. This is because encrypted value is stored inside vertex following write and same vertex is returned (it does not trigger read event as already read).
So I added onAfterUpdate to change back to decrypted value, however whole transaction is written later, meaning it writes unencrypted version. Can provide full JUnit code if needed.

[dijef]

Please check test file I created.
I added boolean USE_EXTRA_AFTER_LISTENERS to show issue I mentioned above when set to true.
EncryptionTest.zip
I think example given can't work, so changes at serialisation would be much more reliable. Please let me know if you can do them, otherwise I'll try to contribute that (I need to get work approval first).

[lvca]

@dijef I can see how encryption (and compression) would be implemented much better into the serializer. I was thinking of providing a simple listener interface that acts right at the beginning for deserialization, converting the encrypted value into a normal buffer and at the end for serialization: when the buffer is created, before returning is encrypted.

Providing a listener could be the quick solution. The best would be providing the listener and a pluggable implementation (an implementation of the listener interface) that does the job of accepting the algorithm to use, keys, etc. So everybody can just configure and use it.

The next step could be allowing to encrypt/decrypt only specific buckets (by configuration) or even only specific properties (probably overkill).

@dijef if you can find time to draft an implementation it would be awesome.

[pawellhasa]

Hello @lvca , my company approved me the contribution. Please assign to this account then. I'll share how I done it.

[lvca]

Great! Just created a new issue from this: #1659