Sooner or later every developer bumps into a project where there is a task to deal with online documents. If there is a suitable library in the framework at hand, they grab it which is of course advisable. But if there isn't, many might say: "It cannot be as difficult. Just use HTTP or FTP and have fun."
The fun wears off, if the internet connection gets broken, or the product owner suddenly wants to pause the upload process, or the infrastructure moves from one cloud provider to another, or there is more than one programming language on the client side etc.
I decided to rely on the professionals who knows how to setup the process efficiently and my choice fell on the tus project. It promised everything I needed:
- stability even if the internet connection is unstable
- freedom where to deploy the service, be it on my own virtual machine or on Google Cloud or AWS
- responsiveness while uploading the file
- security is admittedly not out of the box, but it is possible to inject it where necessary
- documentation and examples in different programming languages
In this article we start with the official example in Go and proceed with the following steps:
- adopt the example for using it with the Google Cloud
- inject a JWT check
- create a metadata for a file
- create a Java client
Let's start with creating main.go like we have it in the official example on github.com. Since we want to code, I get rid of the comments.
package main
import (
"fmt"
"net/http"
"github.com/tus/tusd/pkg/filestore"
tusd "github.com/tus/tusd/pkg/handler"
)
func main() {
store := filestore.FileStore{ Path: "./uploads" }
composer := tusd.NewStoreComposer()
store.UseIn(composer)
handler, err := tusd.NewHandler(tusd.Config{
BasePath: "/files/",
StoreComposer: composer,
NotifyCompleteUploads: true,
})
if err != nil {
panic(fmt.Errorf("Unable to create handler: %s", err))
}
go func() {
for {
event := <-handler.CompleteUploads
fmt.Printf("Upload %s finished\n", event.Upload.ID)
}
}()
http.Handle("/files/", http.StripPrefix("/files/", handler))
err = http.ListenAndServe(":8080", nil)
if err != nil {
panic(fmt.Errorf("Unable to listen: %s", err))
}
}We make this code executable and test it briefly:
> mkdir -p uploads
> go mod init
> go run .
> echo "this is my owesome data" > test.txt
> curl -i -XPOST http://localhost:8080/files/ \
--data-binary "@test.txt" \
-H "Tus-Resumable: 1.0.0" \
-H "Upload-Offset: 0" \
-H "Upload-Length: 24" \
-H "Content-Type: application/offset+octet-stream"
HTTP/1.1 201 Created
Location: http://localhost:8080/files/dd6c76437675c9d3c40b02cfcdfec3b7
Tus-Resumable: 1.0.0
Upload-Offset: 24
X-Content-Type-Options: nosniff
Date: Fri, 05 Mar 2021 16:14:47 GMT
Content-Length: 0As you can see we created the folder uploads because it is what we have as a store in the code:
store := filestore.FileStore{ Path: "./uploads" }curl in our example receives several headers that are a bare minimum and essential for the successful upload.
Now you will find two files under ./uploads: one with the original content and another with the extension .info and the metadata:
24 5 Mar 16:45 dd6c76437675c9d3c40b02cfcdfec3b7
236 5 Mar 16:45 dd6c76437675c9d3c40b02cfcdfec3b7.infoYou also can download the file with curl back to the original place. Just use the header Location you have got with the response after uploading the file:
> curl http://localhost:8080/files/dd6c76437675c9d3c40b02cfcdfec3b7 > test-downloaded.txt
> cat test-downloaded.txt
this is my owesome dataThat was simple enough, wasn't it?
Now we adopt the example to upload the files to the Google Cloud. I assume that you are familiar with the basic Cloud concepts and omit the part with getting access to it. You need at least a service account key with the permission to read and write files to the bucket. Let us assume you have generated this key and saved it into the file google-credentials.json Put this filename into the environment variable in main.go and create the client to operate with the bucket:
import (
...
"os"
"cloud.google.com/go/storage"
"github.com/tus/tusd/pkg/gcsstore"
)
func main() {
os.Setenv("GOOGLE_APPLICATION_CREDENTIALS", "./google-credentials.json")
ctx := context.Background()
client, err := storage.NewClient(ctx)
if err != nil {
panic(err)
}
defer func() {
if err := client.Close(); err != nil {
fmt.Println(err)
}
}()
...
}We also need to replace the filestore with gcsstore:
store := filestore.FileStore{ Path: "./uploads" }
store := gcsstore.New("my-bucket", &gcsstore.GCSService{
Client: client,
})For the purpose of simplicity create my-bucket through the Google Console like we did it with the folder ./uploads. It must exist before we start testing.
That is all! Let us try again with the same curl -XPOST we tried earlier and see that it works (assuming your cloud key and bucket are intact)
I consider it a better architecture pattern to delegate the JWT check to the API Gateway. However, it is not always possible and not the only right way to authorise a request. Let's see how to do it in our Go program.
Tus package has a convenient way to integrate a middleware in an HTTP handler. We are going to use it for checking the header Authorization.
http.Handle("/files/", http.StripPrefix("/files/", handler))
customHandler := http.StripPrefix("/files/", handler)
handler.Middleware(customHandler)
http.Handle("/files/", http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
err := checkJWT(req.Header.Get("Authorization"))
if err != nil {
w.WriteHeader(http.StatusUnauthorized)
}
customHandler.ServeHTTP(w, req)
}))Since we don't want to tinker with the JWT content, let us suppose that the right JWT must always have the content TrueJWT:
func checkJWT(authorizationHeader string) error {
jwt := strings.TrimLeft(authorizationHeader, "Bearer ")
if jwt != "TrueJWT" {
return fmt.Errorf("Access denied")
}
return nil
}If we try with curl -XPOST again, the request fails with 401 Unauthorized, but by sending the header -H "Authorization: Bearer FalseJWT" with the request it succeeds again.
Tus always creates an .info to every uploaded file. Let's take a closer look to it.
> cat dd6c76437675c9d3c40b02cfcdfec3b7.info | jq .
{
"ID": "dd6c76437675c9d3c40b02cfcdfec3b7",
"Size": 24,
"SizeIsDeferred": false,
"Offset": 0,
"MetaData": {},
"IsPartial": false,
"IsFinal": false,
"PartialUploads": null,
"Storage": {
"Path": "uploads/dd6c76437675c9d3c40b02cfcdfec3b7",
"Type": "filestore"
}
}MetaData is empty, but we can use it to our benefit. We don't normally want to lose the original name of the file or the identity of user who has uploaded the file. Metadata comes to our help where we can inject additional information and store it schemeless as we please.
Apart from a lot of different flags the structure tus.Config{} offers a way to implement callback functions. PreUploadCreateCallback is called before an upload starts and it is an appropriate place to create the MetaData:
handler, err := tusd.NewHandler(tusd.Config{
BasePath: "/files/",
StoreComposer: composer,
NotifyCompleteUploads: true,
PreUploadCreateCallback: func(hook tusd.HookEvent) error {
hook.Upload.MetaData["Filename"] = hook.HTTPRequest.Header.Get("Filename")
return nil
},
})As you probably see the filename has to be sent via curl in a header. Therefore, we put one more header to the POST command:
> curl -XPOST http://localhost:8080/files/ \
--data-binary "@test.txt" \
-H "Tus-Resumable: 1.0.0" \
-H "Upload-Offset: 0" \
-H "Upload-Length: 24" \
-H "Content-Type: application/offset+octet-stream"
-H "Filename: test.txt"That should create a new field in MetaData (see .info):
> cat 1b5f7cc42b4590662b8f2b752b64a544.info | jq .
{
"ID": "1b5f7cc42b4590662b8f2b752b64a544",
"Size": 24,
"SizeIsDeferred": false,
"Offset": 0,
"MetaData": {"Filename":"test.txt"},
"IsPartial": false,
"IsFinal": false,
"PartialUploads": null,
"Storage": {
"Path": "uploads/1b5f7cc42b4590662b8f2b752b64a544",
"Type": "filestore"
}
}Now let us take a look at the other side of the bank: the tus client. I would like to switch to Java to have more fun by using another programming language.
There is a very good example of the file upload on github.com I could get along with. However, there is one topic that has been left out: mandatory headers. In order to get this example working I needed to override one function of TusClient and pass additional headers with the request.
private TusClient tusClient = new TusClient() {
@Override
public void prepareConnection(@NotNull HttpURLConnection connection) {
super.prepareConnection(connection);
// we need this header to make the example working
connection.setRequestProperty("Content-Type", "application/offset+octet-stream");
connection.setRequestProperty("Upload-Offset", "0");
// optionally we can add a JWT as well
connection.setRequestProperty("Authorization", "Bearer TrueJWT")
}
};For the download you can use the standard Java libraries, but do not forget the headers as well:
// urlPath is the content of "Location" header that has been sent back with the response while the file was being uploaded
public void downloadFile(final String urlPath) throws MalformedURLException, IOException {
URL url = new URL(urlPath);
URLConnection connection = url.openConnection();
connection.setRequestProperty("Content-Type", "application/offset+octet-stream");
connection.setRequestProperty("Upload-Offset", "0");
BufferedInputStream in = new BufferedInputStream(connection.getInputStream());
FileOutputStream fout = new FileOutputStream(filename);
final byte data[] = new byte[1024];
int count;
while ((count = in.read(data, 0, 1024)) != -1)
fout.write(data, 0, count);
in.close();
fout.close();
}That is all what you have to know for a smooth start with the tus protocol. If you want to download the Go example, we used here, visit us on github.com.
Happy uploading!