data update UI

[DandelionSprout]

Figuring out that I could add more lists from issues with the new list tag, I came to learn that it has become a monumental undertaking that I can no longer accomplish on my own. After 3 hours of research, and having got to 31 lists and 2 list updates at the time of writing (More may or may not be added to that document later), it seems that I've only scraped the tip of the iceberg, especially when I checked out the Firehol lists.

I personally think that more responsibility should be placed on other people than just you (Collin) and me, but it seems to me that pull requests are a daunting thing for many people. Therefore, I think that a GUI form for submitting lists to FilterLists could've been a pretty important addition to the site. I imagine it could've been something á la in the mockup image below, to make it easy enough to understand for people who may not have much experience with GitHub or coding.

[collinbarrett]

I completely agree. This is something I've kicked around in my head for awhile, but never formalized into an issue. Thanks for doing the mockup.

For fields like "License", it would be best to be a dropdown of the licenses currently in the database rather than a free-form field (with a maybe "other" option that allows adding a new one). (No need to update the mockup.)

The backend of this gets tricky. I don't want a public form to directly write the production database (at least the current tables). We would need to gather submissions in a way that allows myself or another maintainer to approve/review to filter out spam, poor quality submissions. It'll require a fair bit of engineering to automate the handling of submission data to remove the manual labor of updating the json files.

Yeah, love the idea, it's just not a trivial one. But, probably a better use of developer time than copy/pasting into json files (which I certainly don't enjoy).

[DandelionSprout]

The most convenient way as I see it, though not necessarily the easiest way in terms of coding, is if submitting the form would've generated a pull request.

Have the name field correspond to "name", the homepage field to "homeUrl", the E-mail address field to "emailAddress", and so on.

[collinbarrett]

Yep, agreed. Oof. Easier said than done, though. Would require learning how to develop against GitHub's API which is a whole new realm I haven't looked at.

Wish I had more hours in a day. :)

[DandelionSprout]

Now you can support this Wish from https://beerpay.io/collinbarrett/FilterLists/wishes/5b93bd965c3b5105c0652ee0

🍻 Cheers!

(My own note: Apparently this message was auto-posted when I made this into a paid wish on Beerpay.)

[collinbarrett]

Thanks, @DandelionSprout . I would love to add this feature, just trying to decide the best architecture that would be relatively easy to build. Been sick the last few days too, so progress has been slower. Thanks for your continued contributions to the project.

[collinbarrett]

possibly useful: https://jsonforms.io/

[ghost]

https://intr0.cf/formtest/

That site is on Netlify, and it's a very basic version of @DandelionSprout 's mock-up with no customization via CSS (it's using the site's main CSS, hence its lack of decent aesthetics).

I'll spin up a worker in CloudFlare in a few hours so you can play with the recipe I create. I don't foresee any roadblocks.

Edit: here's the worker site: https://shiny-haze-148d.why.workers.dev/
& here's the code:

 async function handleRequest(request) {
 const init = {
   headers: {
     "Content-Type": "text/html;charset=UTF-8",
     "Allow": "GET, POST",
     "Access-Control-Allow-Methods": "GET, POST",
     "X-Frame-Options": "deny",
     "X-XSS-Protection": "1; mode=block",
     "Referrer-Policy": "no-referrer",
     "Cross-Origin-Embedder-Policy": "require-corp",
     "Cross-Origin-Opener-Policy": "same-origin",
     "Cross-Origin-Resource-Policy": "same-origin",
     'Clear-Site-Data': '"cache", "cookies", "storage", "executionContexts"',
     "X-DNS-Prefetch-Control": "off",
     "Cache-Control": "private, no-store",
     "Strict-Transport-Security": "max-age=63072000; includeSubDomains; preload",
     "Content-Security-Policy": "base-uri 'none';block-all-mixed-content;connect-src 'none';default-src 'none';font-src 'none';form-action 'self';frame-ancestors 'none';frame-src 'none';img-src 'none';manifest-src 'none';media-src 'none';object-src 'none';require-trusted-types-for 'script';script-src 'none';style-src 'sha256-Apbj+ZycqPKlGzaGwKfovmMZor/0F3Vigz203aq7PZo=';upgrade-insecure-requests;worker-src 'none'",
     "Feature-Policy": "accelerometer 'none';ambient-light-sensor 'none';autoplay 'none';camera 'none';display-capture 'none';document-domain 'none';encrypted-media 'none';fullscreen 'none';geolocation 'none';gyroscope 'none';layout-animations 'none';legacy-image-formats 'none';magnetometer 'none';microphone 'none';midi 'none';oversized-images 'none';payment 'none';picture-in-picture 'none';publickey-credentials 'none';speaker 'none';sync-xhr 'none';unsized-media 'none';usb 'none';vibrate 'none';vr 'none';wake-lock 'none'",
     "Vary": "Origin",
     "X-Content-Type-Options": "nosniff"
   },
 }
 return new Response(someHTML, init)
}
 addEventListener('fetch', event => {
 return event.respondWith(handleRequest(event.request))
})
 const someHTML = `<!doctype html><meta charset=utf-8><style sha256="sha256-Apbj+ZycqPKlGzaGwKfovmMZor/0F3Vigz203aq7PZo=">body{background:#fff;color:#000;font:400 2rem/2.2 ui-sans,BlinkMacSystemFont,'Fira Sans','Noto Sans',sans-serif;margin-left:3rem;margin-right:3rem}html{box-sizing:border-box;font-size:1vh}p{margin:1.5rem}section{padding:.75rem}</style><title>List Submission Form</title><form method=post><h1>List submission form</h1><p>Required fields are followed by <strong><abbr title=required>*</abbr></strong>.<section><h2>Contact information</h2><p><label for=mail><span>Your email: </span><strong><abbr title=required>*</abbr></strong></label> <input id=mail name=usermail type=email></section><section><h3>List information</h3><p><label for=license><span>License type:</span> <strong><abbr title=required>*</abbr></strong></label> <select id=license name=filterlicense><option value=mit>MIT<option value=bsd>BSD<option value=apache>Apache<option value=gpl3>GPLv3<option value=cc4>CC4.0<option value=other>other</select><p><label for=url><span>Filter URL: </span><strong><abbr title=required>*</abbr></strong></label> <input id=url name=filterurl type=url><p><label for=url><span>Homepage:</span></label> <input id=url name=siteurl type=url></section><section><p><label for=text><span>Additional info: </span><input id=info name=additionalinfo></label></section><section><p><button type=submit>Submit</button></section></form>`

[DandelionSprout]

Seems like a good starting point, although my own coding skills are most likely insufficient to modify and improve it myself (e.g. by adding extra text boxes). Collin should most likely be better at this than I am.

[ghost]

If you tell me everything to include, I'll be glad to do it. Then Collin can take over using his CloudFlare acct.~intr0 Ⓐ Sent from ProtonMail Professional On Mon, Mar 23, 2020 at 22:25, Imre Kristoffer Eilertsen <[email protected]> wrote: Seems like a good starting point, although my own coding skills are most likely insufficient to modify and improve it myself (e.g. by adding extra text boxes). Collin should most likely be better at this than I am. —You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe.

[ghost]

I'll be spinning up a new worker site with each change:

https://wild-disk-b338.why.workers.dev/

async function handleRequest(request) {
  const init = {
    headers: {
      "Content-Type": "text/html;charset=UTF-8",
      "Allow": "GET, POST",
      "Access-Control-Allow-Methods": "GET, POST",
      "X-Frame-Options": "deny",
      "X-XSS-Protection": "1; mode=block",
      "Referrer-Policy": "no-referrer",
      "Cross-Origin-Embedder-Policy": "require-corp",
      "Cross-Origin-Opener-Policy": "same-origin",
      "Cross-Origin-Resource-Policy": "same-origin",
      'Clear-Site-Data': '"cache", "cookies", "storage", "executionContexts"',
      "X-DNS-Prefetch-Control": "off",
      "Cache-Control": "private, no-store",
      "Strict-Transport-Security": "max-age=63072000; includeSubDomains; preload",
      "Content-Security-Policy": "base-uri 'none';block-all-mixed-content;connect-src 'none';default-src 'none';font-src 'none';form-action 'self';frame-ancestors 'none';frame-src 'none';img-src 'none';manifest-src 'none';media-src 'none';object-src 'none';require-trusted-types-for 'script';script-src 'none';style-src 'sha256-Apbj+ZycqPKlGzaGwKfovmMZor/0F3Vigz203aq7PZo=';upgrade-insecure-requests;worker-src 'none'",
      "Feature-Policy": "accelerometer 'none';ambient-light-sensor 'none';autoplay 'none';camera 'none';display-capture 'none';document-domain 'none';encrypted-media 'none';fullscreen 'none';geolocation 'none';gyroscope 'none';layout-animations 'none';legacy-image-formats 'none';magnetometer 'none';microphone 'none';midi 'none';oversized-images 'none';payment 'none';picture-in-picture 'none';publickey-credentials 'none';speaker 'none';sync-xhr 'none';unsized-media 'none';usb 'none';vibrate 'none';vr 'none';wake-lock 'none'",
      "Vary": "Origin",
      "X-Content-Type-Options": "nosniff"
    },
  }
  return new Response(someHTML, init)
}
addEventListener('fetch', event => {
  return event.respondWith(handleRequest(event.request))
})

const someHTML = `<!doctype html>
<meta charset=utf-8>
<style sha256="sha256-Apbj+ZycqPKlGzaGwKfovmMZor/0F3Vigz203aq7PZo=">body{background:#fff;color:#000;font:400 2rem/2.2 ui-sans,BlinkMacSystemFont,'Fira Sans','Noto Sans',sans-serif;margin-left:3rem;margin-right:3rem}html{box-sizing:border-box;font-size:1vh}p{margin:1.5rem}section{padding:.75rem}</style>
<title>List Submission Form</title>
<form method=post>
    <!-- Requires formaction= to define endpoint of form POST -->
    <!-- Right now it goes nowhere -->
    <h1>List submission form</h1>
    <p>Required fields are followed by <strong><abbr title=required>*</abbr></strong>.
    <section>
        <h2>Contact information</h2>
        <p>
            <label for=mail><span>Your email: </span><strong><abbr title=required>*</abbr></strong></label>
            <input id=mail name=emailAddress type=email>
    </section>
    <h3>List information</h3>
    <section>
        <p>
            <label for=text><span>List Description: </span><strong><abbr title=required>*</abbr></strong></label>
            <input id=description name=listDescription>
    </section>
    <section>
        <p>
            <label for=syntax><span>List syntax: </span> <strong><abbr title=required>*</abbr></strong></label>
            <select id=syntax name=listSyntax>
                <option value=hosts>Hosts (localhosts)
                <option value=domains>Domains example.com
                <option value=adblocker>AdBlocker-syntax ||example.com^
                <option value=adguard>AdGuard
                <option value=ublockorigin>uBlock Origin
                <option value=other>Other
            </select>
    </section>
    <section>
        <p>
            <label for=license><span>License type: </span> <strong><abbr title=required>*</abbr></strong></label>
            <select id=license name=licenseId>
                <option value=mit>MIT
                <option value=bsd>BSD
                <option value=apache>Apache
                <option value=gpl3>GPLv3
                <option value=cc4>CC4.0
                <option value=other>Other
            </select>
    </section>
    <section>
        <p>
            <label for=url><span>Filter <em>raw</em> URL: </span><strong><abbr title=required>*</abbr></strong></label>
            <input id=url name=viewUrl type=url>
    </section>
    <section>
        <p>
            <label for=text><span>List language (ex: JP:) </span><strong><abbr title=required>*</abbr></strong></label>
            <input id=language name=listLanguage>
    </section>
    <section>
        <p>
            <label for=url><span>Homepage: </span></label>
            <input id=url name=homeUrl type=url>
    </section>
    <section>
        <p>
            <label for=text><span>Additional info: </span>
            <input id=info name=additionalInfo>
            </label>
    </section>
    <section>
        <p>
            <button type=submit>Submit</button>
    </section>
</form>`

And you're welcome! Forgot to say that last night.

[collinbarrett]

Thanks for exploring this, @mozdevcontrib . The UI seems reasonable. I'm just struggling to understand what happens on form submit. Are you thinking the data goes directly to the production database? We could consider that approach...

Ideally, I'd like to have the form maybe open a GitHub PR with the json modified automatically. Something like how the antd GitHub repo accepts PRs. They have a form here that then generates a PR via (presumably) the GitHub API.

Also, adding new lists would be relatively easy and would be a good first step. But, editing existing lists in the form would be a bit more tricky. We could solve that problem in a phase II, I just want to be thinking about that as we design the solution to add.

[ghost]

You're welcome. I read of it on your blog, actually. So, yes, upon the submission of a form it would be POSTed to an endpoint that you'd have to set up. As for GitHub, I'm not very familiar with what's available on their end, but I'd assume they'd allow you set up a web hook in the repo which could somehow be used as the go-between between form submissions and your repo. As for formatting the info in JSON, without having the ability to see the end results, I'm unable to say with certainty, but I don't see that much of a barrier to using something like  `JSON.stringify(reqBody)` directly in the V8 to take care of it. It may require overriding the `Content-Type` response header to `application/json`, but if so, that'd be the simplest part of it. It's the `JSON.stringify` arg that I've not had experience with; though getting that experience first-hand would just take introducing the argument and modifying the rest of the code where it needs to be. ~intr0 Ⓐ Sent from ProtonMail Professional On Tue, Mar 24, 2020 at 16:38, Collin M. Barrett <[email protected]> wrote: Thanks for exploring this, @mozdevcontrib . The UI seems reasonable. I'm just struggling to understand what happens on form submit. Are you thinking the data goes directly to the production database? We could consider that approach... Ideally, I'd like to have the form maybe open a GitHub PR with the json modified automatically. Something like how the antd GitHub repo accepts PRs. They have a form here that then generates a PR via (presumably) the GitHub API. Also, adding new lists would be relatively easy and would be a good first step. But, editing existing lists in the form would be a bit more tricky. We could solve that problem in a phase II, I just want to be thinking about that as we design the solution to add. —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

[ghost]

I just coded in a fake endpoint - https://example.com/submissions - so now upon pressing the submit button it redirects to a 404 Not Found error page. I need to get something set up in GitLab to really get results.

[ghost]

I've found some promising code to pull out of various projects - json2.js is one - also PHP is a possibility considering the simplicity of using it directly within HTML, though I've an aversion to PHP, perhaps it's unreasonable, but I do have it nonetheless. Talk tomorrow. ☮️~intr0 Ⓐ Sent from ProtonMail Professional On Tue, Mar 24, 2020 at 17:30, Ⓐ ~ intr0 <[email protected]> wrote: You're welcome. I read of it on your blog, actually. So, yes, upon the submission of a form it would be POSTed to an endpoint that you'd have to set up. As for GitHub, I'm not very familiar with what's available on their end, but I'd assume they'd allow you see up a web hook in the repo which could somehow be used as the go-between between form submissions and your repo. As for formatting the info in JSON, without having the ability to see the end results, I'm unable to say with certainty, but I don't see that much of a barrier to using something like `JSON.stringify(reqBody)` directly in the V8 to take care of it. It may require overriding the `Content-Type` response header to `application/json`, but if so, that'd be the simplest part of it. It's the `JSON.stringify` arg that I've not had experience with; though getting that experience first-hand would just take introducing the argument and modifying the rest of the code where it needs to be.~intr0 Ⓐ Sent from ProtonMail Professional On Tue, Mar 24, 2020 at 16:38, Collin M. Barrett <[email protected]> wrote: Thanks for exploring this, @mozdevcontrib . The UI seems reasonable. I'm just struggling to understand what happens on form submit. Are you thinking the data goes directly to the production database? We could consider that approach... Ideally, I'd like to have the form maybe open a GitHub PR with the json modified automatically. Something like how the antd GitHub repo accepts PRs. They have a form here that then generates a PR via (presumably) the GitHub API. Also, adding new lists would be relatively easy and would be a good first step. But, editing existing lists in the form would be a bit more tricky. We could solve that problem in a phase II, I just want to be thinking about that as we design the solution to add. —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

[ghost]

I've gone ahead and asked for advice here:

https://community.cloudflare.com/t/sending-submitted-form-data-from-workers-site/158836?u=intr0

I'll check back in the next 24 hours, although I suspect the requested-for advice will come in far more quickly.

[ghost]

That's on https://zapier.com/ There are 235 more GitHub integrations as well. Thought it might be useful.

[collinbarrett]

oops. didn't mean to auto-close this. definitely not done yet. :)