Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package: Analytics Engine #399

Open
wants to merge 29 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
6319887
first up
CraigglesO Oct 1, 2022
e6351c5
remove TODOs; adjust readmes
CraigglesO Oct 1, 2022
6eac0a4
migrate to execute sql data inside engine code
CraigglesO Oct 1, 2022
3ae1bfd
parse INTERVAL; add test cases
CraigglesO Oct 2, 2022
da11a07
interval, complete test cases
CraigglesO Oct 2, 2022
b307341
add miniflare api access + tests; drop 'UNIQUE'
CraigglesO Oct 2, 2022
a5d2fee
fix options; all analytics exists inside singular db now
CraigglesO Oct 2, 2022
cc1547d
add jest testings
CraigglesO Oct 2, 2022
5fb6b54
add jest testings
CraigglesO Oct 2, 2022
b2810df
fix get->all; add http testing in minfilare; remove unique
CraigglesO Oct 2, 2022
11b2692
add vitest tests
CraigglesO Oct 2, 2022
13916f2
fix internal problem
CraigglesO Oct 2, 2022
d798479
TODATETIME fix
CraigglesO Oct 2, 2022
9b359ac
support QUANTILEWEIGHTED
CraigglesO Oct 3, 2022
39ed5cf
support QUANTILEWEIGHTED
CraigglesO Oct 3, 2022
7cb5d40
add formatting; minor fixes/adjustments
CraigglesO Oct 3, 2022
556d5eb
import sorting fix
CraigglesO Oct 3, 2022
16b28f0
more error cases
CraigglesO Oct 4, 2022
7abffeb
re-arrange
CraigglesO Oct 4, 2022
8e1e310
writeDataPoint is sync
CraigglesO Oct 4, 2022
44dc3d8
minor fixes
CraigglesO Oct 4, 2022
c25b639
ensure added functions & keywords are working
CraigglesO Oct 5, 2022
5865e9e
temporarily edit npx-import to get passing tests
CraigglesO Oct 6, 2022
62f63d5
pre-add better-sqlite
CraigglesO Oct 6, 2022
55b1d93
tmp move npx-import
CraigglesO Oct 6, 2022
ca5fe4f
ensure TODATETIME sticks to UTC
CraigglesO Oct 6, 2022
aabd9fb
update to latest master
CraigglesO Oct 24, 2022
c9b272d
first fix set
CraigglesO Oct 24, 2022
f93f54e
bug fixes
CraigglesO Oct 24, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
parse INTERVAL; add test cases
  • Loading branch information
CraigglesO committed Oct 2, 2022
commit 3ae1bfd8fb0581e4ab69182f9d8d4e82b326ca08
2 changes: 1 addition & 1 deletion ava.config.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ export default {
nonSemVerExperiments: {
nextGenConfig: true,
},
files: ["packages/*/test/**/*.spec.ts"],
files: ["packages/analytics-engine/test/**/*.spec.ts"],
timeout: "5m",
nodeArguments: ["--no-warnings", "--experimental-vm-modules"],
typescript: {
Expand Down
30 changes: 28 additions & 2 deletions packages/analytics-engine/src/engine.ts
Original file line number Diff line number Diff line change
Expand Up @@ -106,8 +106,8 @@ export class AnalyticsEngine {
constructor(name: string, db: SqliteDB) {
this.#name = name;
this.#db = db;
buildSQLFunctions(db);
db.exec(analytics.replaceAll("{{BINDING}}", name));
buildSQLFunctions(db);
}

async writeDataPoint({
Expand Down Expand Up @@ -168,7 +168,7 @@ export class AnalyticsEngine {
blobsValues.push(`@${key}`);
});

const insert = this.#db.prepare(
const input = prepare(
`INSERT INTO ${this.#name} (dataset, index1${
doublesKeys.length > 0 ? `, ${doublesKeys}` : ""
}${
Expand All @@ -177,7 +177,33 @@ export class AnalyticsEngine {
doublesValues.length > 0 ? `, ${doublesValues}` : ""
}${blobsValues.length > 0 ? `, ${blobsValues}` : ""})`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like an SQL injection vulnerability. 🙁 Rather than passing values in directly, could you insert placeholders and then fill in the actual values in the this.#db.prepare() call? This also means blob values aren't affected by the replaceAll calls below?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused on this if you don't mind expounding. I'm only inserting blob1, blob2, ..., @blob1, @blob2, ... and the same for doubles depending upon length as far as I am aware.

);
const insert = this.#db.prepare(input);

insert.run(insertData);
}
}

/** @internal */
export function prepare(input: string): string {
// split
const pieces = input.split(" ");
// find all instances of "INTERVAL"
const intervalIndexes = [];
for (let i = 0, pl = pieces.length; i < pl; i++) {
if (pieces[i].toLocaleLowerCase() === "interval") {
intervalIndexes.push(i);
}
}
// for each instance, convert "INTERVAL X Y" to "INTERVAL(X, Y)"
for (const intervalIndex of intervalIndexes) {
const [interval, value, type] = pieces.slice(
intervalIndex,
intervalIndex + 3
);
pieces[intervalIndex] = `${interval}(`;
pieces[intervalIndex + 1] = `${value.replaceAll("'", "")},`;
pieces[intervalIndex + 2] = `'${type.replaceAll("'", "").toUpperCase()}')`;
}

return pieces.join(" ");
}
25 changes: 14 additions & 11 deletions packages/analytics-engine/src/functions.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
import { SqliteDB } from "@miniflare/shared";

export type IntervalTypes =
| "SECOND"
| "MINUTE"
| "HOUR"
| "DAY"
| "MONTH"
| "YEAR";
const TIME = {
SECOND: 1,
MINUTE: 60, // 60sec
HOUR: 60 * 60, // 60sec * 60min
DAY: 60 * 60 * 24, // 60sec * 60min * 24hours
MONTH: 60 * 60 * 24 * 30, // 60sec * 60min * 24hours
YEAR: 60 * 60 * 24 * 365,
};

export default function buildSQLFunctions(sqliteDB: SqliteDB) {
// https://clickhouse.com/docs/en/sql-reference/aggregate-functions/reference/quantileexactweighted/
Expand Down Expand Up @@ -55,13 +56,15 @@ export default function buildSQLFunctions(sqliteDB: SqliteDB) {
// https://clickhouse.com/docs/en/sql-reference/data-types/special-data-types/interval
sqliteDB.function(
"INTERVAL",
(intervalValue: string | number, IntervalTypes: string): number => {
(
intervalValue: string | number,
IntervalType: keyof typeof TIME
): number => {
if (typeof intervalValue === "string") {
intervalValue = parseInt(intervalValue);
}
console.log("NUMBER", intervalValue);
console.log("TYPE", IntervalTypes);
return 0;
const multiplier = TIME[IntervalType] ?? 0;
return intervalValue * multiplier;
}
);
}
Expand Down
42 changes: 35 additions & 7 deletions packages/analytics-engine/test/functions.spec.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { AnalyticsEngine } from "@miniflare/analytics-engine";
import { AnalyticsEngine, prepare } from "@miniflare/analytics-engine";
import { Storage } from "@miniflare/shared";
import { testClock } from "@miniflare/shared-test";
import { MemoryStorage } from "@miniflare/storage-memory";
Expand Down Expand Up @@ -90,10 +90,38 @@ test("Analytics Engine: Test each function to ensure they work.", async (t) => {
const res8 = stmt8.get("a3cd45");
t.true(isDate(res8.answer));

// // test INTERVAL
// const stmt9 = sqliteDB.prepare(
// "SELECT INTERVAL 42 DAY AS answer FROM TEST_BINDING WHERE index1 = ?"
// );
// const res9 = stmt9.get("a3cd45");
// t.true(isDate(res9.answer));
// test INTERVAL
const stmt9Input = prepare(
"SELECT INTERVAL 42 DAY AS answer FROM TEST_BINDING WHERE index1 = ?"
);
const stmt9 = sqliteDB.prepare(stmt9Input);
const res9 = stmt9.get("a3cd45");
t.is(res9.answer, 42 * 60 * 60 * 24);

// test INTERVAL with comments
const stmt10Input = prepare(
"SELECT INTERVAL '42' DAY AS answer FROM TEST_BINDING WHERE index1 = ?"
);
const stmt10 = sqliteDB.prepare(stmt10Input);
const res10 = stmt10.get("a3cd45");
t.is(res10.answer, 42 * 60 * 60 * 24);

// test INTERVAL with comments 2
const stmt11Input = prepare(
"SELECT INTERVAL 42 'DAY' AS answer FROM TEST_BINDING WHERE index1 = ?"
);
const stmt11 = sqliteDB.prepare(stmt11Input);
const res11 = stmt11.get("a3cd45");
t.is(res11.answer, 42 * 60 * 60 * 24);

// test INTERVAL with comments 3
const stmt12Input = prepare(
"SELECT INTERVAL '42 DAY' AS answer FROM TEST_BINDING WHERE index1 = ?"
);
const stmt12 = sqliteDB.prepare(stmt12Input);
const res12 = stmt12.get("a3cd45");
t.is(res12.answer, 42 * 60 * 60 * 24);

// TODO: test SECOND, MINUTE, ...
// TODO: test if not the right word
});