Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Create directory #26

Merged
merged 10 commits into from
Oct 6, 2016
226 changes: 197 additions & 29 deletions web/files/files.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,11 @@
package files

import (
"errors"
"fmt"
"io"
"net/http"
"regexp"
"strings"

"github.com/cozy/cozy-stack/web/jsonapi"
Expand All @@ -19,54 +22,159 @@ import (
// DefaultContentType is used for files uploaded with no content-type
const DefaultContentType = "application/octet-stream"

// Upload is the handler for uploading a file
//
// swagger:route POST /files/:folder-id files uploadFile
// DocType is the type of document, eg. file or folder
type DocType string

const (
// FileDocType is document type
FileDocType DocType = "io.cozy.files"
// FolderDocType is document type
FolderDocType = "io.cozy.folders"
)

var (
errDocAlreadyExists = errors.New("Directory already exists")
errDocTypeInvalid = errors.New("Invalid document type")
errIllegalFilename = errors.New("Invalid filename: empty or contains one of these illegal characters: / \\ : ? * \" |")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How did you choose what chars are legal / illegal?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By looking at Windows and its naming conventions... https://msdn.microsoft.com/en-us/library/aa365247(v=vs.85).aspx (I'm missing > and < btw)...

Don't you think it may cause us problem if one day we want to sync on a win fs ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least, we should forbid the null character (\0).

For sync with windows, there are things more difficult than just some invalid chars. Some filename are reserved (AUX, COM1, etc), the length of the path is limited (260 chars on NTFS), the last char can't be a space or a dot, it is not case sensitive, etc.

I don't think we should enforce all these rules on Cozy. For the legal/illegal chars, I don't mind. I was just curious how you choosed them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't want to enforce these rules, I don't think its necessary to forbid these special characters either. I'll just leave / as an illegal one for now.

New PR: #30

)

var regFileName = regexp.MustCompile("[\\/\\\\:\\?\\*\"|]+")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's necessary to escape /, ? and * in this regexp.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, and it may be simpler to use strings.IndexAny to check for forbidden chars

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, strings.ContainsAny is even more appropriate.


// DocMetadata encapsulates the few metadata linked to a document
// creation request.
type DocMetadata struct {
Type DocType
Name string
FolderID string
Executable bool
Tags []string
}

func (metadata *DocMetadata) path() string {
return metadata.FolderID + "/" + metadata.Name
}

// NewDocMetadata is the DocMetadata constructor. All inputs are
// validated and if wrong, an error is returned.
func NewDocMetadata(docTypeStr, name, folderID, tagsStr string, executable bool) (*DocMetadata, error) {
docType, err := parseDocType(docTypeStr)
if err != nil {
return nil, err
}

if err = checkFileName(name); err != nil {
return nil, err
}

// FolderID is not mandatory. If empty, the document is at the root
// of the FS
if folderID != "" {
if err = checkFileName(folderID); err != nil {
return nil, err
}
}

tags := parseTags(tagsStr)

return &DocMetadata{
Type: docType,
Name: name,
FolderID: folderID,
Tags: tags,
Executable: executable,
}, nil
}

// Upload is the method for uploading a file
//
// This will be used to upload a file
func Upload(c *gin.Context) {
doctype := c.Query("Type")
if doctype != "io.cozy.files" {
err := fmt.Errorf("Invalid Type in the query-string")
c.AbortWithError(422, err)
return
// @TODO
func Upload(metadata *DocMetadata, storage afero.Fs, body io.ReadCloser) error {
if metadata.Type != FileDocType {
return errDocTypeInvalid
}

path := metadata.path()

defer body.Close()
return afero.SafeWriteReader(storage, path, body)
}

// CreateDirectory is the method for creating a new directory
//
// @TODO
func CreateDirectory(metadata *DocMetadata, storage afero.Fs) error {
if metadata.Type != FolderDocType {
return errDocTypeInvalid
}

name := c.Query("Name")
if name == "" {
err := fmt.Errorf("Missing Name in the query-string")
c.AbortWithError(422, err)
path := metadata.path()

exists, err := afero.DirExists(storage, path)
if err != nil {
return err
}
if exists {
return errDocAlreadyExists
}

return storage.Mkdir(path, 0777)
}

// CreationHandler handle all POST requests on /files/:folder-id
// aiming at creating a new document in the FS. Given the Type
// parameter of the request, it will either upload a new file or
// create a new directory.
//
// swagger:route POST /files/:folder-id files uploadFileOrCreateDir
func CreationHandler(c *gin.Context) {
instance := middlewares.GetInstance(c)
storage, err := instance.GetStorageProvider()
if err != nil {
c.AbortWithError(http.StatusInternalServerError, err)
return
}

tags := strings.Split(c.Query("Tags"), ",")
executable := c.Query("Executable") == "true"
folderID := c.Param("folder-id")
metadata, err := NewDocMetadata(
c.Query("Type"),
c.Query("Name"),
c.Param("folder-id"),
c.Query("Tags"),
c.Query("Executable") == "true",
)

if err != nil {
c.AbortWithError(http.StatusUnprocessableEntity, err)
return
}

contentType := c.ContentType()
if contentType == "" {
contentType = DefaultContentType
}

instanceInterface, exists := c.Get("instance")
if !exists {
err := fmt.Errorf("No instance found")
exists, err := checkParentFolderID(storage, metadata.FolderID)
if err != nil {
c.AbortWithError(http.StatusInternalServerError, err)
return
}
instance := instanceInterface.(*middlewares.Instance)
storage, err := instance.GetStorageProvider()
if err != nil {
c.AbortWithError(http.StatusInternalServerError, err)
if !exists {
err = fmt.Errorf("Parent folder with given FolderID does not exist")
c.AbortWithError(http.StatusNotFound, err)
return
}

fmt.Printf("%s:\n\t- %s\n\t- %v\n\t- %v\n", name, contentType, tags, executable)
path := folderID + "/" + name
err = afero.SafeWriteReader(storage, path, c.Request.Body)
c.Request.Body.Close()
fmt.Printf("%s:\n\t- %+v\n\t- %v\n", metadata.Name, metadata, contentType)

switch metadata.Type {
case FileDocType:
err = Upload(metadata, storage, c.Request.Body)
case FolderDocType:
err = CreateDirectory(metadata, storage)
}

if err != nil {
c.AbortWithError(http.StatusInternalServerError, err)
c.AbortWithError(makeCode(err), err)
return
}

Expand All @@ -76,5 +184,65 @@ func Upload(c *gin.Context) {

// Routes sets the routing for the files service
func Routes(router *gin.RouterGroup) {
router.POST("/:folder-id", Upload)
router.POST("/", CreationHandler)
router.POST("/:folder-id", CreationHandler)
}

func makeCode(err error) (code int) {
switch err {
case errDocAlreadyExists:
code = http.StatusConflict
default:
code = http.StatusInternalServerError
}
return
}

func parseTags(str string) []string {
var tags []string
for _, tag := range strings.Split(str, ",") {
// @TODO: more sanitization maybe ?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can remove this TODO. I don't see a reason to restrict what the tags can be

tag = strings.TrimSpace(tag)
if tag != "" {
tags = append(tags, tag)
}
}
return tags
}

func parseDocType(docType string) (DocType, error) {
var result DocType
var err error
switch docType {
case "io.cozy.files":
result = FileDocType
case "io.cozy.folders":
result = FolderDocType
default:
err = errDocTypeInvalid
}
return result, err
}

func checkFileName(str string) error {
if str == "" || regFileName.MatchString(str) {
return errIllegalFilename
}
return nil
}

func checkParentFolderID(storage afero.Fs, folderID string) (bool, error) {
if folderID == "" {
return true, nil
}

exists, err := afero.DirExists(storage, folderID)
if err != nil {
return false, err
}
if !exists {
return false, nil
}

return true, nil
}
54 changes: 53 additions & 1 deletion web/files/upload_test.go → web/files/files_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,60 @@ func injectInstance(instance *middlewares.Instance) gin.HandlerFunc {
}
}

func createDir(t *testing.T, path string) *http.Response {
res, err := http.Post(ts.URL+path, "text/plain", strings.NewReader(""))
assert.NoError(t, err)
return res
}

func upload(t *testing.T, path, contentType, body string) *http.Response {
buf := strings.NewReader(body)
res, err := http.Post(ts.URL+path, contentType, buf)
assert.NoError(t, err)
return res
}

func TestCreateDirWithNoType(t *testing.T) {
res := createDir(t, "/files/123")
assert.Equal(t, 422, res.StatusCode)
res.Body.Close()
}

func TestCreateDirWithNoName(t *testing.T) {
res := createDir(t, "/files/123?Type=io.cozy.folders")
assert.Equal(t, 422, res.StatusCode)
res.Body.Close()
}

func TestCreateDirOnNonExistingParent(t *testing.T) {
res := createDir(t, "/files/noooop?Name=foo&Type=io.cozy.folders")
assert.Equal(t, 404, res.StatusCode)
res.Body.Close()
}

func TestCreateDirAlreadyExists(t *testing.T) {
res := createDir(t, "/files/123?Name=456&Type=io.cozy.folders")
assert.Equal(t, 409, res.StatusCode)
res.Body.Close()
}

func TestCreateDirSuccess(t *testing.T) {
res := createDir(t, "/files/123?Name=coucou&Type=io.cozy.folders")
assert.Equal(t, 201, res.StatusCode)
res.Body.Close()

storage, _ := instance.GetStorageProvider()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess gometalinter wants you to factorize this part of the code with TestCreateDirWithSlashInName into a testDirExist(t, instance, expectedPath) function

exists, err := afero.DirExists(storage, "123/coucou")
assert.NoError(t, err)
assert.True(t, exists)
}

func TestCreateDirWithIllegalCharacter(t *testing.T) {
res := createDir(t, "/files/123?Name=coucou/les/copains!&Type=io.cozy.folders")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure we want to allow creation of a folder with "/" in names as it will not work when mapping it to an actual FS (like cozy-desktop does). We should probalby just throw a 400 "no slash in name".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed. Made changes to reject illegal characters in 3ffba18

assert.Equal(t, 422, res.StatusCode)
res.Body.Close()
}

func TestUploadWithNoType(t *testing.T) {
res := upload(t, "/files/123", "text/plain", "foo")
assert.Equal(t, 422, res.StatusCode)
Expand Down Expand Up @@ -59,9 +106,14 @@ func TestMain(m *testing.M) {
Domain: "test",
StorageURL: "mem:https://test",
}

storage, _ := instance.GetStorageProvider()
storage.Mkdir("123", 0777)
storage.Mkdir("123/456", 0777)

router := gin.New()
router.Use(injectInstance(instance))
router.POST("/files/:folder-id", Upload)
router.POST("/files/:folder-id", CreationHandler)
ts = httptest.NewServer(router)
defer ts.Close()
os.Exit(m.Run())
Expand Down
6 changes: 6 additions & 0 deletions web/middlewares/instance.go
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,9 @@ func SetInstance() gin.HandlerFunc {
c.Set("instance", instance)
}
}

// GetInstance will return the instance linked to the given gin
// context or panic if none exists
func GetInstance(c *gin.Context) *Instance {
return c.MustGet("instance").(*Instance)
}