Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Produce OpenAPI document describing CCF's endpoints #1612

Merged
merged 64 commits into from
Sep 25, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
6f7df55
Add to_upper and to_lower to nonstd.h
eddyashton Jun 24, 2020
af0ef44
Use nonstd::to_lower
eddyashton Jun 24, 2020
78e5e78
WIP: Add types for OpenAPI doc
eddyashton Jun 24, 2020
94d3779
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Jun 25, 2020
2f1a965
WIP: "api" returns openapi, not method list
eddyashton Jun 25, 2020
1510aaf
Prefer openapi to openapi_schema
eddyashton Jun 25, 2020
fb3466a
WIP: Maybe a deadend...
eddyashton Jun 25, 2020
ed846b7
WIP: 3 cheers for operator[]
eddyashton Jun 25, 2020
32b9d46
Is this what a helpful modifier looks like?
eddyashton Jun 25, 2020
325a59c
WIP: Helper method for adding response schema
eddyashton Jun 25, 2020
8ad27d2
Add schema_name to generate sensible components fieldname
eddyashton Jun 26, 2020
bd1eb15
WIP
eddyashton Jun 26, 2020
0ef2951
WIP: add_request_body_schema
eddyashton Jun 26, 2020
964f863
WIP: Forgive me father for I have sinned
eddyashton Jun 29, 2020
b508e87
WIP: Support optional fields
eddyashton Jun 29, 2020
89340ce
Fix for optional fields
eddyashton Jun 29, 2020
da7039e
WIP: Add support for renames
eddyashton Jun 29, 2020
016c7df
WIP: Fix duplicate-required and missing-base-elements
eddyashton Jun 29, 2020
7a8ebd3
Support for vectors, fancy types
eddyashton Jun 30, 2020
1b9fc33
Remove some fixed TODOs
eddyashton Jun 30, 2020
e92bb37
Separate simple and complex test cases
eddyashton Jun 30, 2020
593005d
Steps towards pure JSON doc
eddyashton Jun 30, 2020
a01d2c5
Partial compile fixes
eddyashton Sep 9, 2020
d4b3a6e
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Sep 9, 2020
b7e75ea
Get to compilation
eddyashton Sep 9, 2020
55a4f73
Update schema.py to just validate that the result is a valid OpenAPI doc
eddyashton Sep 9, 2020
9c6adcc
Remove separate schema. Oh no
eddyashton Sep 9, 2020
8f736e3
Progress: schema_test dumps OpenAPI docs to file, initial files
eddyashton Sep 10, 2020
0296f09
Produce schema that the online validator is happy with
eddyashton Sep 10, 2020
1b24113
Drop an element the spec validator isn't happy with
eddyashton Sep 10, 2020
55d9402
Use freeform JSON manipulators everywhere, fix nested add_to_components
eddyashton Sep 10, 2020
d439221
Schema.py should continue to retrieve individual schema
eddyashton Sep 11, 2020
39de645
And here's those schemas again
eddyashton Sep 11, 2020
7314eb2
Strip out some uninteresting TODOs
eddyashton Sep 11, 2020
afc0999
Add schema for templated endpoints too
eddyashton Sep 11, 2020
45dc53d
WIO
eddyashton Sep 11, 2020
f74258d
Compile fixes
eddyashton Sep 14, 2020
669c1f6
Safe capture
eddyashton Sep 15, 2020
a4dcb9d
Prefix OpenAPI paths
eddyashton Sep 15, 2020
2ee7336
Get things in-sync
eddyashton Sep 15, 2020
6cdeab9
Progress
eddyashton Sep 15, 2020
ddf38bd
Split out params on all endpoints
eddyashton Sep 15, 2020
0970c09
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Sep 15, 2020
ff405dd
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Sep 15, 2020
57c214a
Format
eddyashton Sep 15, 2020
0701c77
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Sep 16, 2020
c28ace5
Delete
eddyashton Sep 16, 2020
7596241
Oops - these lines restore result schema
eddyashton Sep 16, 2020
8d5a1b8
Add restored result schema
eddyashton Sep 16, 2020
78905b5
Document (minimum) JS endpoints
eddyashton Sep 17, 2020
08fe5a6
Hey we can do that in lua too
eddyashton Sep 17, 2020
890c9d7
Shout about TODOs again
eddyashton Sep 17, 2020
89d4d73
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Sep 17, 2020
86889e8
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Sep 21, 2020
c31c300
Fixups
eddyashton Sep 21, 2020
90c0c08
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Sep 22, 2020
2132beb
Merge branch 'openapi_generation2' of github.com:eddyashton/CCF into …
eddyashton Sep 22, 2020
7e63db4
Make OpenAPI info customisable
eddyashton Sep 23, 2020
7c9b986
Update supports_method
eddyashton Sep 23, 2020
9241b61
Format
eddyashton Sep 23, 2020
c3353fa
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Sep 24, 2020
56cf98b
Use is_arithmetic
eddyashton Sep 24, 2020
54c2c6c
Don't print in test
eddyashton Sep 24, 2020
4142618
Merge branch 'master' into openapi_generation2
eddyashton Sep 24, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
WIP: "api" returns openapi, not method list
  • Loading branch information
eddyashton committed Jun 25, 2020
commit 2f1a9658d7e8e951b0865ae1e3e5804f95f93bac
17 changes: 9 additions & 8 deletions src/apps/js_generic/js_generic.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -285,15 +285,16 @@ namespace ccfapp

// Since we do our own dispatch within the default handler, report the
// supported methods here
void list_methods(kv::Tx& tx, ListMethods::Out& out) override
void build_api(ds::openapi::Document& document, kv::Tx& tx) override
{
UserEndpointRegistry::list_methods(tx, out);

auto scripts = tx.get_view(this->network.app_scripts);
scripts->foreach([&out](const auto& key, const auto&) {
out.methods.push_back(key);
return true;
});
UserEndpointRegistry::build_api(document, tx);

// TODO
// auto scripts = tx.get_view(this->network.app_scripts);
// scripts->foreach([&out](const auto& key, const auto&) {
// out.methods.push_back(key);
// return true;
// });
}
};

Expand Down
23 changes: 12 additions & 11 deletions src/apps/lua_generic/lua_generic.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -189,18 +189,19 @@ namespace ccfapp

// Since we do our own dispatch within the default handler, report the
// supported methods here
void list_methods(kv::Tx& tx, ListMethods::Out& out) override
void build_api(ds::openapi::Document& document, kv::Tx& tx) override
{
UserEndpointRegistry::list_methods(tx, out);

auto scripts = tx.get_view(this->network.app_scripts);
scripts->foreach([&out](const auto& key, const auto&) {
if (key != UserScriptIds::ENV_HANDLER)
{
out.methods.push_back(key);
}
return true;
});
UserEndpointRegistry::build_api(document, tx);

// TODO
// auto scripts = tx.get_view(this->network.app_scripts);
// scripts->foreach([&out](const auto& key, const auto&) {
// if (key != UserScriptIds::ENV_HANDLER)
// {
// out.methods.push_back(key);
// }
// return true;
// });
}
};

Expand Down
5 changes: 1 addition & 4 deletions src/ds/test/openapi_schema.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -16,18 +16,15 @@ TEST_CASE("Basic doc")
using namespace ds;
openapi::Document doc;
doc.info.title = "Test generated API";
doc.info.description = "Some longer description enhance with **Markdown**";
doc.info.description = "Some longer description enhanced with **Markdown**";
doc.info.version = "0.1.42";

print_doc("INITIAL", doc);

{
openapi::Server mockup_server;
mockup_server.url =
"https://virtserver.swaggerhub.com/eddyashton/ccf-test/1.0.0";
doc.servers.push_back(mockup_server);
}
print_doc("SERVERS", doc);

{
doc.paths["/users/foo"]
Expand Down
8 changes: 3 additions & 5 deletions src/node/rpc/call_types.h
Original file line number Diff line number Diff line change
Expand Up @@ -117,12 +117,10 @@ namespace ccf
using Out = CallerInfo;
};

struct ListMethods
struct GetAPI
{
struct Out
{
std::vector<std::string> methods;
};
// TODO: This is a ds::openapi::Document?
using Out = nlohmann::json;
};

struct GetSchema
Expand Down
22 changes: 10 additions & 12 deletions src/node/rpc/common_endpoint_registry.h
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,10 @@ namespace ccf

public:
CommonEndpointRegistry(
kv::Store& store, const std::string& certs_table_name = "") :
EndpointRegistry(store, certs_table_name),
const std::string& method_prefix_,
kv::Store& store,
const std::string& certs_table_name = "") :
EndpointRegistry(method_prefix_, store, certs_table_name),
nodes(store.get<Nodes>(Tables::NODES)),
tables(&store)
{}
Expand Down Expand Up @@ -219,17 +221,13 @@ namespace ccf
.set_auto_schema<GetNodesByRPCAddress::In, GetNodesByRPCAddress::Out>()
.install();

auto list_methods_fn = [this](kv::Tx& tx, nlohmann::json&& params) {
ListMethods::Out out;

list_methods(tx, out);

std::sort(out.methods.begin(), out.methods.end());

return make_success(out);
auto openapi = [this](kv::Tx& tx, nlohmann::json&& params) {
ds::openapi::Document document;
build_api(document, tx);
return make_success(document);
};
make_endpoint("api", HTTP_GET, json_adapter(list_methods_fn))
.set_auto_schema<void, ListMethods::Out>()
make_endpoint("api", HTTP_GET, json_adapter(openapi))
.set_auto_schema<void, GetAPI::Out>()
.install();

auto get_schema = [this](auto& args, nlohmann::json&& params) {
Expand Down
26 changes: 17 additions & 9 deletions src/node/rpc/endpoint_registry.h
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

#include "ds/ccf_deprecated.h"
#include "ds/json_schema.h"
#include "ds/openapi_schema.h"
#include "enclave/rpc_context.h"
#include "http/http_consts.h"
#include "kv/store.h"
Expand Down Expand Up @@ -68,6 +69,8 @@ namespace ccf
Write
};

const std::string method_prefix;

/** An Endpoint represents a user-defined resource that can be invoked by
* authorised users via HTTP requests, over TLS. An Endpoint is accessible
* at a specific verb and URI, e.g. POST /app/accounts or GET /app/records.
Expand Down Expand Up @@ -305,7 +308,10 @@ namespace ccf

public:
EndpointRegistry(
kv::Store& tables, const std::string& certs_table_name = "")
const std::string& method_prefix_,
kv::Store& tables,
const std::string& certs_table_name = "") :
method_prefix(method_prefix_)
{
if (!certs_table_name.empty())
{
Expand Down Expand Up @@ -432,17 +438,19 @@ namespace ccf
return default_handler.value();
}

/** Populate out with all supported methods
*
* This is virtual since the default endpoint may do its own dispatch
* internally, so derived implementations must be able to populate the list
* with the supported methods however it constructs them.
*/
virtual void list_methods(kv::Tx& tx, ListMethods::Out& out)
// TODO: May want the entire rpc context, not just tx?
virtual void build_api(ds::openapi::Document& document, kv::Tx& tx)
{
for (const auto& [method, verb_handlers] : installed_handlers)
{
out.methods.push_back(method);
const auto full_path = fmt::format("/{}/{}", method_prefix, method);
auto& path_object = document.paths[full_path];
for (const auto& [verb, handler] : verb_handlers)
{
path_object.operations[verb]
.responses[std::to_string(HTTP_STATUS_OK)]
.description = "Auto-generated";
}
}
}

Expand Down
5 changes: 4 additions & 1 deletion src/node/rpc/member_frontend.h
Original file line number Diff line number Diff line change
Expand Up @@ -549,7 +549,10 @@ namespace ccf
NetworkTables& network,
AbstractNodeState& node,
ShareManager& share_manager) :
CommonEndpointRegistry(*network.tables, Tables::MEMBER_CERTS),
CommonEndpointRegistry(
get_actor_prefix(ActorsType::members),
*network.tables,
Tables::MEMBER_CERTS),
network(network),
node(node),
share_manager(share_manager),
Expand Down
3 changes: 2 additions & 1 deletion src/node/rpc/node_frontend.h
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,8 @@ namespace ccf

public:
NodeEndpoints(NetworkState& network, AbstractNodeState& node) :
CommonEndpointRegistry(*network.tables),
CommonEndpointRegistry(
get_actor_prefix(ActorsType::nodes), *network.tables),
network(network),
node(node)
{}
Expand Down
3 changes: 1 addition & 2 deletions src/node/rpc/serialization.h
Original file line number Diff line number Diff line change
Expand Up @@ -108,8 +108,7 @@ namespace ccf
DECLARE_JSON_TYPE(GetUserId::In)
DECLARE_JSON_REQUIRED_FIELDS(GetUserId::In, cert)

DECLARE_JSON_TYPE(ListMethods::Out)
DECLARE_JSON_REQUIRED_FIELDS(ListMethods::Out, methods)
DECLARE_JSON_TYPE(GetAPI::Out)

DECLARE_JSON_TYPE(GetSchema::In)
DECLARE_JSON_REQUIRED_FIELDS(GetSchema::In, method)
Expand Down
2 changes: 1 addition & 1 deletion src/node/rpc/test/frontend_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,7 @@ class TestNoCertsFrontend : public RpcFrontend
public:
TestNoCertsFrontend(kv::Store& tables) :
RpcFrontend(tables, endpoints),
endpoints(tables)
endpoints("test", tables)
{
open();

Expand Down
5 changes: 3 additions & 2 deletions src/node/rpc/user_frontend.h
Original file line number Diff line number Diff line change
Expand Up @@ -81,11 +81,12 @@ namespace ccf
{
public:
UserEndpointRegistry(kv::Store& store) :
CommonEndpointRegistry(store, Tables::USER_CERTS)
CommonEndpointRegistry(
get_actor_prefix(ActorsType::users), store, Tables::USER_CERTS)
{}

UserEndpointRegistry(NetworkTables& network) :
CommonEndpointRegistry(*network.tables, Tables::USER_CERTS)
UserEndpointRegistry(*network.tables)
{}
};

Expand Down
1 change: 1 addition & 0 deletions tests/schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ def fetch_schema(client):
check(
list_response, error=lambda status, msg: status == http.HTTPStatus.OK.value
)
LOG.warning(list_response)
methods = list_response.result["methods"]

for method in methods:
Expand Down