Add OPA project proposal

[caniszczyk]

http:https://www.openpolicyagent.org/

Signed-off-by: Chris Aniszczyk [email protected]

[caniszczyk]

FYI @kenowens12 thanks for the original proposal, I removed it from the repo and made it into a PR, we shouldn't merge in project proposals until they are approved by the @cncf/toc

[jessfraz]

The PRINCIPLES.md doc says that projects should "Has users, preferably in production; is a high quality, high-velocity project." Does this project fulfill that requirement? I can't find any links to active users on the website.

[tsandall]

Thanks @caniszczyk for submitting this.

hey @jessfraz! We've listed a few adopters in the proposal. E.g., Netflix uses OPA to enforce authorization policies across HTTP and gRPC APIs, Kafka, SSH, and so on. We presented this work yesterday: https://kccncna17.sched.com/event/CU73.

[jessfraz]

As a very new project is there a specific reason you need to be in the foundation. What are you hoping to get out of it? What is your project struggling with?

[tsandall]

The most important thing the foundation can provide is a vendor-neutral, well respected home for the project. We've had several early stage/potential adopters ask why the project is NOT part of the foundation. When we started talking with various folks from CNCF (e.g., TOC members, contributors, project leads, etc.) we found we were well aligned on the mission statement and felt qualified for the inception stage. We think the project fills an important gap in the ecosystem around policy and authorization and we see OPA as being complementary to many projects in the space. So it seems like a great fit.

[bgrant0607]

We should clarify in the principles doc that the "preferably in production" clause is most applicable to incubation-level projects. Inception is targeted at earlier-stage projects.

[emaildanwilson]

I'm excited to see this as a possible inception project!

I think this will solve a whole range of problems for companies as adoption increases (as well as complexity). Let's face it, with all those CRD's to manage we'll need detailed control over a large set of object types. IMO, we should encourage a common way of doing this across the growing ecosystem of APIs.

[jessfraz]

curious what you all think about the SecPAL design from a few years ago and how you designed yours? are there any design docs you all have open to the public?

[johnbelamaric]

I think that this is an important part of the ecosystem. In fact we have built a similar open source engine since we needed it before OPA was ready. By externalizing policy from a specific application (like Kubernetes) you can keep the decision making simple in those applications. Complex, very narrow use cases can still be handled even without extending the core application's code to handle them. You just have to install the right policies, and those policies can be modified any time to adjust to new requirements.

Before moving forward with our own engine, we did a survey of what was out there (my philosophy being to use what's available whenever possible). There was nothing lightweight and embeddable in a Go program that offered the kind of flexibility we needed, except OPA. Unfortunately this was nearly a year ago and before OPA was ready for our use. If we were evaluating it now we may have made a different choice.

One of the goals of the CNCF is to promote cloud native architectures, and that includes the development of a reference architecture that includes the major components needed to build an effective platform. Right now, there is no place in the high-level architecture from the TOC nor in the landscape that captures the concept of an externalized policy engine. Adding OPA to the CNCF will encourage the development of hooks into existing CNCF projects to integrate with externalized policy engines, and provides a CNCF project to make this somewhat abstract niche more concrete.

[tim-styra]

@jessfraz I haven’t heard someone mention SecPal for years--good reference! SecPal is a Datalog variant that makes delegation easy; it’s syntactic sugar on top of Datalog. OPA is actually a generalization of Datalog (everything you can write in Datalog you can write in OPA, but not vice versa). OPA was designed as Datalog adapted to support JSON data (instead of just relational data), putting it somewhere between Datalog and Prolog in terms of expressiveness (which is why the syntax is similar to Datalog/Prolog today).

If you’re into formal semantics for policy languages, I dug out some slides we put together for a presentation with the Stanford Logic group a year or so ago and put them up here. Be warned--it’s just the math.

https://docs.google.com/presentation/d/11REkmh94FNbPUU67GJQXlXu0C9duqrW1HsLbg2TuC5U/edit?usp=sharing

We’ve put most of our energy into keeping our actual docs up to date instead of keeping a design doc up to date. But here’s the first one we put together (this was even before Torin started on the project).

https://docs.google.com/document/d/1JSOmiKUr5ZEIT61wgDtGHvz2vgOrtumJrECEzHBqJW0/edit#heading=h.177zfbgphuj3

The more recent design proposal spearheaded by the Google folks is in the OPA repo.

https://github.com/open-policy-agent/opa/blob/master/design/REGO_V2_PROPOSAL.md

[tim-styra]

One thing I overstated there. OPA doesn't support recursion today (whereas traditional Datalog does). Because of the way we're using packages and simple builtins like walk(), we can get a long way without recursion.

[jessfraz]

Thanks so much for the informative reply, this is super cool :)

It might be cool to add this stuff on your website, or maybe I’m just a weirdo for wondering hahaha unsure who the audience would be :)

[monadic]

+1 what @jessfraz said. The datalog provenance is helpful.

[ghost]

I've reviewed the proposal, and discussed it with Torin and Tim, and am strongly supportive. We are very excited about the proposed addition of OPA to the CNCF. Having available a CNCF-owned, well supported policy language and runtime will make it significantly easier and more attractive for CNCF projects to optionally integrate with a consistent policy expression and enforcement mechanism and common toolset around that.

[jbeda]

One thing that I'd love to see here is a separation of the API used to call OPA and the implementation itself.

Specifically, I'd love to have a SubjectAccessReview definition that systems beyond OPA can support. This would enable a thriving ecosystem of policy implementations and a way to leverage plumbing in to various systems.

As a point of reference, see the SAR webhook implementation for k8s -- https://kubernetes.io/docs/admin/authorization/webhook/.

Just as there is a split between SPIFFE and SPIRE, I would love to see a separation of "Subject Access Review API" and OPA where OPA is one way to service that API.

[josephjacks]

+1 from me on @jbeda's comments.. was going to respond similarly.

Over the next year or so, I think we are going to see a significant increased focus on language-level abstraction. With that said, I am a bit concerned about the scope of Rego v2 (in extending CEL) beyond its policy DSL -- i.e. maybe adding future support for recursion, Turing-completeness, etc.

If OPA is about policy enforcement, shouldn't Rego strictly be a DSL for policy enforcement? Maybe I'm missing something obvious. Surely we don't need another Turing-complete language to enforce policy?

I think decoupling OPA from Rego in this proposal would make it easier for current and future language-level tools in the ecosystem to interact with OPA and thereby minimize integration/extensibility/interoperability friction and/or politics. My 2¢!

[tsandall]

Down the road we'd be happy to work with the community to standardize an API. So far, we've focused on making sure OPA is flexible enough to integrate with existing integration-points without requiring modification to OPA itself. For example, you can use OPA as a webhook authorizer or an external admission controller in Kubernetes today. In those cases, the policy has access to the entire SubjectAccessReview (or AdmissionReview) and is responsible for generating the entire response message body--OPA itself is not coupled to either of these domain/project-specific data models (it's all just JSON data.) We've also done integrations with Terraform, Docker, Istio, Linkerd, CloudFoundry, and PAM. We want a bunch of experience integrating with different types of systems before we start trying to define a standard API for integration.

/cc @jbeda

[tsandall]

The changes introduced in the V2 proposal are largely syntactic (the semantics are unchanged.) They’re mostly aimed at improving usability of the language. We have no plans of making the language Turing complete (it’s actually a strict non-goal.)

/cc @josephjacks

[TristonianJones]

As an FYI, there is an effort to setup a CNCF working group for policy to discuss API proposals. I've had some early discussions with a few stakeholders about it and there is a draft document about Safe Access For Everyone (SAFE) which speaks more to these efforts that I would be happy to share.

I view OPA and policy APIs much like MySQL and JDBC. MySQL is an implementation of a database that works well for many use cases. JDBC ensures that the details of connecting and calling MySQL are abstracted away. However, there are many JDBC compatible databases and many interfaces that can be used to connect to MySQL. OPA is a great policy engine, language, toolset that works well for many policy implementations, and it should be possible to integrate into many (more) interfaces for writing and managing policy.

/cc @jbeda

[jessfraz]

I think the point is to make it explicit that the API is a spec that can be implemented by anyone else and is not owned by a specific implementation. So as not to run into the same problem as Java.

…

On Wed, Jan 10, 2018 at 5:20 PM, Tristan Swadell ***@***.***> wrote: As an FYI, there is an effort to setup a CNCF working group for policy to discuss API proposals. I've had some early discussions with a few stakeholders about it and there is a draft document about Safe Access For Everyone (SAFE) which speaks more to these efforts that I would be happy to share. I view OPA and policy APIs much like MySQL and JDBC. MySQL is an implementation of a database that works well for many use cases. JDBC ensures that the details of connecting and calling MySQL are abstracted away. However, there are many JDBC compatible databases and many interfaces that can be used to connect to MySQL. OPA is a great policy engine, language, toolset that works well for many policy implementations, and it should be possible to integrate into many (more) interfaces for writing and managing policy. /cc @jbeda <https://github.com/jbeda> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#71 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABYNbAEmwRgaZtOwwdMqF1JLBgojAlDHks5tJTetgaJpZM4Q2cVv> .

-- Jessie Frazelle 4096R / D4C4 DD60 0D66 F65A 8EFC 511E 18F3 685C 0022 BFF3 pgp.mit.edu <http:https://pgp.mit.edu/pks/lookup?op=get&search=0x18F3685C0022BFF3>

[TristonianJones]

I'm really thrilled about the interests and asks on this thread. I definitely appreciate and understand the desire for policy standards (including APIs). Your points are well-taken @jessfraz, @jbeda, @josephjacks.

OPA is a good source of inspiration with respect to standards, but it is not an authorization API (even though it can be used for API authorization!). If OPA as an implementation increases the desire for such APIs in a subsequent CNCF proposal, I would consider that a testament to its strength as an incubation candidate.

In my own opinion, I believe there will always be many ways to express policy. This thinking is also evident in the http:https://padme.io project. A working group should establish semantic / performance / usability guidelines for different use cases and benchmarks / tests created to help developers understand which solution is best for them. When it comes to policy standards, I'm interested in advancing the following conversations, though I think they are beyond the scope of OPA:

• Deployment - policy release workflow and versioning
• Lifecycle - associating policy with a subject / target (and its lifecycle)
• Enforcement - common patterns for deriving context relevant to policy and APIs for enforcing it
• Correctness - methods for verifying correctness / standardizing testability
• Auditing - point in time access testing, report generation

[erinboyd]

+1 for inception. The community could benefit from a broad policy based engine and language. Given it's inception phase, it can mature through the community and concerns about seperation from language to engine can be addressed there

[joshuamckenty]

+1 for inception. The lack of an external policy library and grammar has, from my perspective, held the entire Cloud Native community back from engaging with critical stakeholders. OPA is the first candidate in this space that seems to strike the right balance of flexibility and usability.

[ultrasaurus]

could someone wrap the text at 80 columns?

[ghost]

@ultrasaurus The other proposals don't follow any particular convention, and I don't think that's really necessary. If you'd like to look at a nicely formatted version, click the "view" button, or go here?

[ultrasaurus]

@quinton-hoole it doesn't really bother me, just seemed to be a requirement activities elsewhere -- I thought it was a CNCF practice, but maybe that's isolated to Serverless WG specification work.

[ultrasaurus]

I just re-read the proposal and realize that the context for the Google collaboration on this proposal might not be clear. Google has a widely used, large scale implementation of security policy that is consistent with the proposed next version of OPA's Rego v2: Firebase Rules. Firebase recently surpassed 1 Million developers. Firebase Rules allows for direct-from-mobile access to Firebase's back-end-as-a-service APIs, so that mobile and Web developers can write client-side only code to develop applications. (For more detail on use cases, see Serverless WG White Paper Mobile Backends

@TristonianJones is the Team Lead for Firebase Rules and has been collaborating on the Rego v2 proposal for most of the past year. This new iteration of the OPA policy language supports all use cases for Google's internal proprietary implementation of Firebase Rules, as well as additional requirements from other Google products. Our goal is to have Firebase Rules interoperate with OPA with the aim that developers can secure their cloud native apps with policy that is expressed consistently with other parts of their infrastructure.

[bgrant0607]

@quinton-hoole The main reason to line-wrap is to enable commenting on lines rather than just whole paragraphs.

[caniszczyk]

@bgrant0607 has agreed to be the additional TOC sponsor as part of the new sandbox process, we will work with the OPA team to update the proposal and finalize things by the next TOC meeting on March 20th

[justincormack]

I spent some time building a prototype to get a feel for OPA and found it very easy to use and flexible. The integration as a Go library is really easy, and it would be a huge benefit if applications were to move away from ad-hoc tooling for policies, and we standardised on a single layer that we could build higher level tooling around.

[caniszczyk]

final call for any more @cncf/toc sponsors outside of the 2 minimum we have for the sandbox, if I don't hear from anyone by next Monday, we will have OPA enter the sandbox

[mayakacz]

Excited about this, inception would be great.
In particular we're lacking a way to express transportable security policies, initial readthroughs seems like this could help.
A few questions:

• is anyone using OPA already for enforcing security policies?
• do you foresee any limits on the 'decisions' that OPA can make? I assume the actual enforcement would be up to the service(?), so if a service is acting badly, OPA has no way of forcing it to act a certain way?

[tsandall]

@mayakacz a number of organizations are using OPA to enforce security policies in different parts of their stack.

This talk from the last KubeCon explains how Netflix uses OPA to enforce authorization policies in Kafka, HTTP and gRPC APIs, and so on. OPA is also being used to enforce admission control policies in Kubernetes.

Ultimately, OPA can't guarantee that the service enforces the decision. That being said, when OPA is integrated into frameworks like Spring or platforms like Istio it becomes less of an issue.

[tamalsaha]

@tsandall , I tried to learn about OPA this weekend from http:https://www.openpolicyagent.org/docs/how-does-opa-work.html#the-data-document doc.

It seems to me that OPA can identify violations in a input json doc. But do you one define spec for right behavior using OPA rego language? Say I was to write Role, ClusterRole or AuditPolicy specs in Kubernetes using OPA?

If I am already using tools that use HCL to define policy, how does it compare to those?

[tsandall]

@tamalsaha OPA doesn't assign any special meaning to violations. It's just a rule that generates a set of objects (which represent servers violating the example policy.) You could rename violations to my_bad_servers and OPA would not care. The only thing that would need to be updated is the service that queries OPA, i.e., it would need to query for my_bad_servers instead of violations.

With OPA you define rules that generate decisions. Decisions are represented as JSON values like objects, arrays, booleans, strings, etc. If you wanted to use OPA to generate objects like Roles or RoleBindings, you could.

HCL is a configuration language, so it's goals are somewhat different. OPA focuses on letting you express decision logic required in policies. With OPA you can write policies that search and join over collections of data. AFAIK, HCL only has limited support for looping.

If you have other questions, feel free to ask on Slack.

[suniljames]

Many congrats to the OPA community and Styra for helping shepherd this to life. Well deserved!

[caniszczyk]

https://www.cncf.io/blog/2018/03/29/cncf-to-host-open-policy-agent-opa/