Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Produce OpenAPI document describing CCF's endpoints #1612

Merged
merged 64 commits into from
Sep 25, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
6f7df55
Add to_upper and to_lower to nonstd.h
eddyashton Jun 24, 2020
af0ef44
Use nonstd::to_lower
eddyashton Jun 24, 2020
78e5e78
WIP: Add types for OpenAPI doc
eddyashton Jun 24, 2020
94d3779
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Jun 25, 2020
2f1a965
WIP: "api" returns openapi, not method list
eddyashton Jun 25, 2020
1510aaf
Prefer openapi to openapi_schema
eddyashton Jun 25, 2020
fb3466a
WIP: Maybe a deadend...
eddyashton Jun 25, 2020
ed846b7
WIP: 3 cheers for operator[]
eddyashton Jun 25, 2020
32b9d46
Is this what a helpful modifier looks like?
eddyashton Jun 25, 2020
325a59c
WIP: Helper method for adding response schema
eddyashton Jun 25, 2020
8ad27d2
Add schema_name to generate sensible components fieldname
eddyashton Jun 26, 2020
bd1eb15
WIP
eddyashton Jun 26, 2020
0ef2951
WIP: add_request_body_schema
eddyashton Jun 26, 2020
964f863
WIP: Forgive me father for I have sinned
eddyashton Jun 29, 2020
b508e87
WIP: Support optional fields
eddyashton Jun 29, 2020
89340ce
Fix for optional fields
eddyashton Jun 29, 2020
da7039e
WIP: Add support for renames
eddyashton Jun 29, 2020
016c7df
WIP: Fix duplicate-required and missing-base-elements
eddyashton Jun 29, 2020
7a8ebd3
Support for vectors, fancy types
eddyashton Jun 30, 2020
1b9fc33
Remove some fixed TODOs
eddyashton Jun 30, 2020
e92bb37
Separate simple and complex test cases
eddyashton Jun 30, 2020
593005d
Steps towards pure JSON doc
eddyashton Jun 30, 2020
a01d2c5
Partial compile fixes
eddyashton Sep 9, 2020
d4b3a6e
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Sep 9, 2020
b7e75ea
Get to compilation
eddyashton Sep 9, 2020
55a4f73
Update schema.py to just validate that the result is a valid OpenAPI doc
eddyashton Sep 9, 2020
9c6adcc
Remove separate schema. Oh no
eddyashton Sep 9, 2020
8f736e3
Progress: schema_test dumps OpenAPI docs to file, initial files
eddyashton Sep 10, 2020
0296f09
Produce schema that the online validator is happy with
eddyashton Sep 10, 2020
1b24113
Drop an element the spec validator isn't happy with
eddyashton Sep 10, 2020
55d9402
Use freeform JSON manipulators everywhere, fix nested add_to_components
eddyashton Sep 10, 2020
d439221
Schema.py should continue to retrieve individual schema
eddyashton Sep 11, 2020
39de645
And here's those schemas again
eddyashton Sep 11, 2020
7314eb2
Strip out some uninteresting TODOs
eddyashton Sep 11, 2020
afc0999
Add schema for templated endpoints too
eddyashton Sep 11, 2020
45dc53d
WIO
eddyashton Sep 11, 2020
f74258d
Compile fixes
eddyashton Sep 14, 2020
669c1f6
Safe capture
eddyashton Sep 15, 2020
a4dcb9d
Prefix OpenAPI paths
eddyashton Sep 15, 2020
2ee7336
Get things in-sync
eddyashton Sep 15, 2020
6cdeab9
Progress
eddyashton Sep 15, 2020
ddf38bd
Split out params on all endpoints
eddyashton Sep 15, 2020
0970c09
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Sep 15, 2020
ff405dd
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Sep 15, 2020
57c214a
Format
eddyashton Sep 15, 2020
0701c77
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Sep 16, 2020
c28ace5
Delete
eddyashton Sep 16, 2020
7596241
Oops - these lines restore result schema
eddyashton Sep 16, 2020
8d5a1b8
Add restored result schema
eddyashton Sep 16, 2020
78905b5
Document (minimum) JS endpoints
eddyashton Sep 17, 2020
08fe5a6
Hey we can do that in lua too
eddyashton Sep 17, 2020
890c9d7
Shout about TODOs again
eddyashton Sep 17, 2020
89d4d73
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Sep 17, 2020
86889e8
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Sep 21, 2020
c31c300
Fixups
eddyashton Sep 21, 2020
90c0c08
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Sep 22, 2020
2132beb
Merge branch 'openapi_generation2' of github.com:eddyashton/CCF into …
eddyashton Sep 22, 2020
7e63db4
Make OpenAPI info customisable
eddyashton Sep 23, 2020
7c9b986
Update supports_method
eddyashton Sep 23, 2020
9241b61
Format
eddyashton Sep 23, 2020
c3353fa
Merge branch 'master' of github.com:microsoft/CCF into openapi_genera…
eddyashton Sep 24, 2020
56cf98b
Use is_arithmetic
eddyashton Sep 24, 2020
54c2c6c
Don't print in test
eddyashton Sep 24, 2020
4142618
Merge branch 'master' into openapi_generation2
eddyashton Sep 24, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Get to compilation
  • Loading branch information
eddyashton committed Sep 9, 2020
commit b7e75ea309a8872e2938aa053f3a6ac4b19275f5
4 changes: 2 additions & 2 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -253,9 +253,9 @@ if(BUILD_TESTS)
)

add_unit_test(
openapi ${CMAKE_CURRENT_SOURCE_DIR}/src/ds/test/openapi.cpp
openapi_test ${CMAKE_CURRENT_SOURCE_DIR}/src/ds/test/openapi.cpp
)
target_link_libraries(openapi PRIVATE http_parser.host)
target_link_libraries(openapi_test PRIVATE http_parser.host)

add_unit_test(
logger_json_test
Expand Down
2 changes: 1 addition & 1 deletion src/apps/js_generic/js_generic.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -584,7 +584,7 @@ namespace ccfapp

// Since we do our own dispatch within the default handler, report the
// supported methods here
void build_api(ds::openapi::Document& document, kv::Tx& tx) override
void build_api(nlohmann::json& document, kv::Tx& tx) override
{
UserEndpointRegistry::build_api(document, tx);

Expand Down
2 changes: 1 addition & 1 deletion src/apps/lua_generic/lua_generic.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@ namespace ccfapp

// Since we do our own dispatch within the default handler, report the
// supported methods here
void build_api(ds::openapi::Document& document, kv::Tx& tx) override
void build_api(nlohmann::json& document, kv::Tx& tx) override
{
UserEndpointRegistry::build_api(document, tx);

Expand Down
17 changes: 11 additions & 6 deletions src/ds/json.h
Original file line number Diff line number Diff line change
Expand Up @@ -549,7 +549,7 @@ namespace std
fill_json_schema_required_fields(j, t); \
POST_FILL_SCHEMA; \
} \
inline std::string schema_name(const TYPE& t) \
inline std::string schema_name(const TYPE&) \
{ \
return #TYPE; \
} \
Expand Down Expand Up @@ -599,31 +599,36 @@ namespace std
add_schema_components_optional_fields(doc, j, t))

#define DECLARE_JSON_REQUIRED_FIELDS(TYPE, ...) \
inline void to_json_required_fields(nlohmann::json& j, const TYPE& t) \
inline void to_json_required_fields( \
nlohmann::json& j, [[maybe_unused]] const TYPE& t) \
{ \
if (!j.is_object()) \
{ \
j = nlohmann::json::object(); \
} \
_FOR_JSON_COUNT_NN(__VA_ARGS__)(POP1)(WRITE_REQUIRED, TYPE, ##__VA_ARGS__) \
} \
inline void from_json_required_fields(const nlohmann::json& j, TYPE& t) \
inline void from_json_required_fields( \
const nlohmann::json& j, [[maybe_unused]] TYPE& t) \
{ \
if (!j.is_object()) \
{ \
throw JsonParseError("Expected object, found: " + j.dump()); \
} \
_FOR_JSON_COUNT_NN(__VA_ARGS__)(POP1)(READ_REQUIRED, TYPE, ##__VA_ARGS__) \
} \
inline void fill_json_schema_required_fields(nlohmann::json& j, const TYPE&) \
inline void fill_json_schema_required_fields( \
nlohmann::json& j, [[maybe_unused]] const TYPE& t) \
{ \
j["type"] = "object"; \
_FOR_JSON_COUNT_NN(__VA_ARGS__) \
(POP1)(FILL_SCHEMA_REQUIRED, TYPE, ##__VA_ARGS__) \
} \
template <typename T> \
void add_schema_components_required_fields( \
T& doc, nlohmann::json& j, const TYPE& t) \
[[maybe_unused]] T& doc, \
nlohmann::json& j, \
[[maybe_unused]] const TYPE& t) \
{ \
j["type"] = "object"; \
_FOR_JSON_COUNT_NN(__VA_ARGS__) \
Expand Down Expand Up @@ -681,7 +686,7 @@ namespace std
} \
template <typename T> \
void add_schema_components_optional_fields( \
T& doc, nlohmann::json& j, const TYPE& t) \
T& doc, nlohmann::json& j, const TYPE&) \
{ \
_FOR_JSON_COUNT_NN(__VA_ARGS__) \
(POP1)(ADD_SCHEMA_COMPONENTS_OPTIONAL, TYPE, ##__VA_ARGS__); \
Expand Down
4 changes: 2 additions & 2 deletions src/ds/nonstd.h
Original file line number Diff line number Diff line change
Expand Up @@ -64,14 +64,14 @@ namespace nonstd

/** converts strings to upper or lower case, in-place
*/
static void to_upper(std::string& s)
static inline void to_upper(std::string& s)
{
std::transform(s.begin(), s.end(), s.begin(), [](unsigned char c) {
return std::toupper(c);
});
}

static void to_lower(std::string& s)
static inline void to_lower(std::string& s)
{
std::transform(s.begin(), s.end(), s.begin(), [](unsigned char c) {
return std::tolower(c);
Expand Down
41 changes: 27 additions & 14 deletions src/ds/openapi.h
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@
#include <nlohmann/json.hpp>
#include <string>

#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wgnu-zero-variadic-macro-arguments"

namespace ds
{
/**
Expand All @@ -21,20 +24,22 @@ namespace ds
{
namespace access
{
static nlohmann::json& get_object(nlohmann::json& j, const std::string& k)
static inline nlohmann::json& get_object(
nlohmann::json& j, const std::string& k)
{
const auto ib = j.emplace(k, nlohmann::json::object());
return ib.first.value();
}

static nlohmann::json& get_array(nlohmann::json& j, const std::string& k)
static inline nlohmann::json& get_array(
nlohmann::json& j, const std::string& k)
{
const auto ib = j.emplace(k, nlohmann::json::array());
return ib.first.value();
}
}

static nlohmann::json create_document(
static inline nlohmann::json create_document(
const std::string& title,
const std::string& description,
const std::string& document_version)
Expand All @@ -49,29 +54,39 @@ namespace ds
{"paths", nlohmann::json::object()}};
}

static nlohmann::json& server(nlohmann::json& document, const std::string& url)
static inline nlohmann::json& server(
nlohmann::json& document, const std::string& url)
{
auto& servers = access::get_object(document, "servers");
servers.push_back({{"url", url}});
return servers.back();
}

static nlohmann::json& path(nlohmann::json& document, const std::string& path)
static inline nlohmann::json& path(
nlohmann::json& document, const std::string& path)
{
// TODO: Check that path starts with /?
auto& paths = access::get_object(document, "paths");
return access::get_object(paths, path);
}

static nlohmann::json& path_operation(nlohmann::json& path, http_method verb)
static inline nlohmann::json& path_operation(
nlohmann::json& path, http_method verb)
{
// HTTP_GET becomes the string "get"
std::string s = http_method_str(verb);
nonstd::to_lower(s);
return access::get_object(path, s);
}

static nlohmann::json& response(
static inline nlohmann::json& path_operation(
nlohmann::json& path, std::string s)
{
nonstd::to_lower(s);
return access::get_object(path, s);
}

static inline nlohmann::json& response(
nlohmann::json& path_operation,
http_status status,
const std::string& description = "Default response description")
Expand All @@ -85,19 +100,15 @@ namespace ds
return response;
}

static nlohmann::json& request_body(
nlohmann::json& path_operation
)
static inline nlohmann::json& request_body(nlohmann::json& path_operation)
{
auto& request_body = access::get_object(path_operation, "requestBody");
access::get_object(request_body, "content");
return request_body;
}

static nlohmann::json& media_type(
nlohmann::json& j,
const std::string& mt
)
static inline nlohmann::json& media_type(
nlohmann::json& j, const std::string& mt)
{
auto& content = access::get_object(j, "content");
return access::get_object(content, mt);
Expand Down Expand Up @@ -402,3 +413,5 @@ namespace ds
DECLARE_JSON_OPTIONAL_FIELDS(Document, servers, components);
}
}

#pragma clang diagnostic pop
12 changes: 1 addition & 11 deletions src/node/rpc/call_types.h
Original file line number Diff line number Diff line change
Expand Up @@ -134,18 +134,8 @@ namespace ccf

struct GetAPI
{
struct Endpoint
{
std::string verb;
std::string path;
};

// TODO: What is the format of this now?
//using Out = nlohmann::json;
struct Out
{
std::vector<Endpoint> endpoints;
};
using Out = nlohmann::json;
};

struct EndpointMetrics
Expand Down
10 changes: 5 additions & 5 deletions src/node/rpc/common_endpoint_registry.h
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,8 @@ namespace ccf
.install();

auto openapi = [this](kv::Tx& tx, nlohmann::json&&) {
ds::openapi::Document document;
// TODO: Update these
auto document = ds::openapi::create_document("Placeholder title", "Placeholder description", "1.0.0");
build_api(document, tx);
return make_success(document);
};
Expand Down Expand Up @@ -276,10 +277,9 @@ namespace ccf

if (j.empty())
{
std::string verb_name = http_method_str(verb);
nonstd::to_lower(verb_name);
j[verb_name] =
GetSchema::Out{endpoint.params_schema, endpoint.result_schema};
return make_error(
HTTP_STATUS_BAD_REQUEST,
fmt::format("Method {} not recognised", in.method));
}

return make_success(j);
Expand Down
15 changes: 10 additions & 5 deletions src/node/rpc/endpoint_registry.h
Original file line number Diff line number Diff line change
Expand Up @@ -449,15 +449,20 @@ namespace ccf
* with the supported endpoints however it defines them.
*/
// TODO: May want the entire rpc context, not just tx?
virtual void build_api(ds::openapi::Document& document, kv::Tx& tx)
virtual void build_api(nlohmann::json& document, kv::Tx&)
{
for (const auto& [path, verb_endpoints] : fully_qualified_endpoints)
{
const auto full_path = fmt::format("/{}/{}", method_prefix, method);
auto& path_object = document.paths[full_path];
for (const auto& [verb, handler] : verb_handlers)
const auto full_path = fmt::format("/{}/{}", method_prefix, path);
auto& path_object = ds::openapi::path(document, "/users/foo");
for (const auto& [verb, handler] : verb_endpoints)
{
path_object[verb][HTTP_STATUS_OK].description = "Auto-generated";
auto& path_operation = ds::openapi::path_operation(path_object, verb.c_str());
auto& path_response_ok = ds::openapi::response(
path_operation, HTTP_STATUS_OK, "Auto-generated");
auto& path_response_ok_json = ds::openapi::media_type(
path_response_ok, http::headervalues::contenttype::JSON);
path_response_ok_json["schema"] = "Placeholder";
}
}

Expand Down
6 changes: 0 additions & 6 deletions src/node/rpc/serialization.h
Original file line number Diff line number Diff line change
Expand Up @@ -112,12 +112,6 @@ namespace ccf
DECLARE_JSON_TYPE(GetUserId::In)
DECLARE_JSON_REQUIRED_FIELDS(GetUserId::In, cert)

//DECLARE_JSON_TYPE(GetAPI::Out)
DECLARE_JSON_TYPE(ListMethods::Endpoint)
DECLARE_JSON_REQUIRED_FIELDS(ListMethods::Endpoint, verb, path)
DECLARE_JSON_TYPE(ListMethods::Out)
DECLARE_JSON_REQUIRED_FIELDS(ListMethods::Out, endpoints)

DECLARE_JSON_TYPE(EndpointMetrics::Metric)
DECLARE_JSON_REQUIRED_FIELDS(EndpointMetrics::Metric, calls, errors, failures)
DECLARE_JSON_TYPE(EndpointMetrics::Out)
Expand Down