diff --git a/TODO.txt b/TODO.txt index 2628d75..defa910 100644 --- a/TODO.txt +++ b/TODO.txt @@ -1,25 +1,19 @@ -Projet: -- [ ] faire retourner l armee au village -- [ ] enlever les RawXXX -- [ ] + de cohesion avec les persist -- [ ] securite verifier que les actions peuvent seulement etre faites par la bonne personne -- [ ] test coverage -- [ ] avant de constuire un bulding verifier assez argent et or -- [ ] get village qui retourne aussi les buildings -- [*] Tester avec GPA pour les memory leaks (shutdown et sigint a mettre dans doc si ca marche) - Doc: -- [ ] Ecrire doc du projet +- [ ] get village qui retourne aussi les buildings +- [ ] finir api design +- [?] CI - [ ] TOUT relire (verifier que tous les premiers scripts sont executables) Fin: +- [ ] rmettre db schema a jour - [ ] sources zotero -- [ ] faire un autre export ou autre branche ou je mets tout joli pour word ou latex avec les bonnes images, titres, liens etc - [ ] corriger fautes orthographe - [ ] Affiche - [ ] Resume publiable Peut etre: +- [?] securite verifier que les actions peuvent seulement etre faites par la bonne personne +- [*] Tester avec GPA pour les memory leaks (shutdown et sigint a mettre dans doc si ca marche) - [?] + traiter les erreurs et moins de try - [?] zig interop avec rust ? - [?] Un script pour lancer les serveurs se serait pas mal @@ -42,3 +36,6 @@ Futur: Questions: - est ce que je peux rajouter qqch d interessant pour la concurence par exmple pour epoll, psq je trouve assez fade? +- voir pour zotero comment faire les liens +- besoin d avoir "liste des codes, figures, ..." ? +- voir ensemble la partie projet diff --git a/content/docs/project-1/auth.org b/content/docs/project-1/auth.org index 40a3b50..9a2add6 100644 --- a/content/docs/project-1/auth.org +++ b/content/docs/project-1/auth.org @@ -18,6 +18,5 @@ If the user lost his cookie or connect on a new device, he will provide its =use *** POST /auth/logout The user will access this endpoint with no needed paramater as he is already authentified by the session token. The server will then delete the session token from the database and from the user's cookie. - -TODO montrer avec du code et expliquer cookies pas secure - rendre cookies secure Secure; HttpOnly; SameSite=Strict +*** Security issues +Since it is not a core concept of my Bachelor Thesis I did not spend too much time on the security aspects so obivously there are a lot of things that could be better like adding =Secure=, =HttpOnly= and =SameSite=Strict= to the cookies. I also did not add any salt or pepper to the hashing of the password. And a few other details as well. diff --git a/content/docs/project-1/conclusion.org b/content/docs/project-1/conclusion.org index 541f3e5..618914f 100644 --- a/content/docs/project-1/conclusion.org +++ b/content/docs/project-1/conclusion.org @@ -3,48 +3,32 @@ #+hugo_cascade_type: docs #+math: true -The [[https://github.com/Pismice/Zig-Conquest][repository of the project]] is available on GitHub and open-source for now, might not be in the future for security reasons. - -TODO parler des avantages et inconvenients de zig pour ce projet -TODO rentable de prendre zig pour perf alors que db bottleneck ou pas ? faire un petit benchmark - -* SQLite Zig wrapper -Issues with chaining dynamic requests -#+BEGIN_SRC zig - pub fn createBuilding(db: *sqlite.Db, comptime BuildingType: type, building: *BuildingType) !void { - // 1. Create the building "parent" - switch (BuildingType) { - Building.GoldMine => |_| { - const query = - \\ INSERT INTO buildings(level,space_taken) VALUES(1,0); - \\ INSERT INTO gold_mines(building_id,productivity) VALUES(last_insert_rowid(),?) - ; - const gm: *GoldMine = @ptrCast(building); - var stmt = try db.prepare(query); - defer stmt.deinit(); - try stmt.exec(.{}, .{ .productivity = gm.productivity }); - }, - else => return error.UnkownBuildingType, - } - } -#+END_SRC - -#+begin_src zig -pub fn createBuilding(db: *sqlite.Db, comptime BuildingType: type, building: *BuildingType) !void { - // 1. Create the building "parent" - - var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator); - defer arena.deinit(); - const allocator = arena.allocator(); - - // 2. Create the building "child" - switch (BuildingType) { - Building.GoldMine => |_| { - const gm: *GoldMine = @ptrCast(building); - const query = try std.fmt.allocPrint(allocator, "INSERT INTO buildings(level,space_taken) VALUES(1,0);INSERT INTO gold_mines(building_id,productivity) VALUES(last_insert_rowid(),{d})", .{gm.productivity}); - try db.execMulti(query, .{}); - }, - else => return error.UnkownBuildingType, - } -} -#+end_src +The [[https://github.com/Pismice/Zig-Conquest][repository of the project]] is available on GitHub and open-source for now, it might not be in the future for security reasons. + +The project is still evolving so all the documentation here might not always be up to date. Note that at the time of writing this I still do not have a decend frontend solution, I am using only [[https://www.postman.com/][Postman]] for the moment to test the API. + +It is hard to say if Zig was a good choice for this project programmer experience wise it is can really be relative to every people but still this language on this point has some good and bad points. The first very good point of Zig is that compared to C it is the nicest possible experience, especially in the context of a web server where there are a lot of dynamic allocations happening at multiple places, having allocators here truly simplified the developpment process, the framework I used could either sometimes ask me to pass an allocator and handle the memory by myself or sometimes it would provide some allocators for me, I therefore knew that there was no need to worry about this memory space. + +To keep the comparaison going with C, the Zig package manager also is great, you do not have to clone a repository, create a Makefile and so on, everything worked out nicely. + +The =comptime= keyword also inderectly helped me a lot through the sqlite wrapper I used, it could check at compilation time if the bind parameters were correct, which is great for the dev experience, since you run into less runtime errors. + +Even though Zig seem to surpass C in the context of web developpment it still has some major concurents like Rust, Go, Java, C# and so on which are probably more adapted for writing a general case web server. Since Zig is pretty low level there are still a lot of things you have to manage yourself and the environment is not as mature as the other languages I cited. + +To conclude on the dev experience, Zig was a decent choice for this type of project, it made the dev experience a bit harder than with other languages (except C) but since it has some pretty good performances it was a good choice for the context of a video game server. + +Performance wise since almost all my requests have to hit the database, it became my bottleneck. The number of requests I could treat went down dramatically for requests that have to communicate with the database. + +I did a test where I simply did a =SELECT= query on a dozens of rows for each request and ended up with those results. + +#+CAPTION: Transfers per second for different implementations with and without DB +#+NAME: fig:SED-HR4049 +[[/HEIG_ZIG/images/db_perf.png]] + +We can see how the ratio of requests per second went down when I started to hit the database especially for httpz it went down for more than 10 times the number of requests it could handle, but for the std implementation it only went down 2 times. + +I did not give too much attention into why hitting the database made it such a huge bottleneck, but from my common sense and experience I would say that the order of magnitude quite makes sense. But this show that having the fastest possible HTTP server is not always the best choice since the bottleneck might be somewhere else. + +To conclude on the performance, I would say that Zig is a decent choice for a video game server, it has some pretty good performances and if the bottleneck can be reduced it could be a great choice. + +Overall Zig was good choice, I could ease into the project and start to implement features quite quickly, the language is quite nice to use and the performances are good. The fact that the framework was good and easy to use also probably helped. diff --git a/content/docs/project-1/db-schema.org b/content/docs/project-1/db-schema.org index abebc53..bfaec4f 100644 --- a/content/docs/project-1/db-schema.org +++ b/content/docs/project-1/db-schema.org @@ -1,4 +1,4 @@ -#+title: Database schema +#+title: Database #+weight: 3 #+hugo_cascade_type: docs #+math: true @@ -8,6 +8,82 @@ Every =Village= has a =garnison= army to defend the village all the time. The pl Not all the values necessary to make the game work are stored on the database, for example the gold it takes to upgarde a building, the power of each units, etc. Those values are stored in the server code to avoid having to fetch them all the time even though they never change. -#+CAPTION: Transfers per second for different frameworks +#+CAPTION: DB schema #+NAME: fig:SED-HR4049 [[/HEIG_ZIG/images/schema.png]] + +* SQLite Zig wrapper +The library overwall was quite nice to use because of all the benefits I cited in the *Introduction* section of this project. But it still had a little drawback being the lack of support and clear documentations. I often found myself stuck with errors that are not very detailled. + +One of the main issues I had was with [[https://github.com/vrischmann/zig-sqlite/issues/162][chaining dynamic requests]]. + +The following code is not working and returning an =SQLiteRangeError= error. This is because I tried to execute 2 queries inside one. Note that it is important here that the 2 queries are executed one after the other because of the =last_insert_rowid()= function that could not return the wanted id if something else has been done on the database at the same time. +#+BEGIN_SRC zig + pub fn createBuilding(db: *sqlite.Db, comptime BuildingType: type, building: *BuildingType) !void { + // 1. Create the building "parent" + switch (BuildingType) { + Building.GoldMine => |_| { + const query = + \\ INSERT INTO buildings(level,space_taken) VALUES(1,0); + \\ INSERT INTO gold_mines(building_id,productivity) VALUES(last_insert_rowid(),?) + ; + const gm: *GoldMine = @ptrCast(building); + var stmt = try db.prepare(query); + defer stmt.deinit(); + try stmt.exec(.{}, .{ .productivity = gm.productivity }); + }, + else => return error.UnkownBuildingType, + } + } +#+END_SRC + +This can be fixed in two different ways, the first is to not use the library =?= wildcard but instead make my whole query without using those and then calling =execMulti=, note that =execMulti= can not take any =?= parameters. + +#+begin_src zig + pub fn createBuilding(db: *sqlite.Db, comptime BuildingType: type, building: *BuildingType) !void { + // 1. Create the building "parent" + + var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator); + defer arena.deinit(); + const allocator = arena.allocator(); + + // 2. Create the building "child" + switch (BuildingType) { + Building.GoldMine => |_| { + const gm: *GoldMine = @ptrCast(building); + const query = try std.fmt.allocPrint(allocator, "INSERT INTO buildings(level,space_taken) VALUES(1,0);INSERT INTO gold_mines(building_id,productivity) VALUES(last_insert_rowid(),{d})", .{gm.productivity}); + try db.execMulti(query, .{}); + }, + else => return error.UnkownBuildingType, + } + } +#+end_src + +The second and in my opinion best way to do it in order to get both the advantages of executing requests together and also having the possibility to use =?= parameter is to use a =savepoint=. The example below is not from the same code but it still illustrates the point well. + +#+begin_src zig + pub fn createAttackingArmy(self: *Village, db: *sqlite.Db, allocator: std.mem.Allocator, attackInfos: Troops) !*Army { + // Verify that he has enough units in his village + const source_village_army = try self.getArmy(db, allocator); + if (source_village_army.nb_ranged < attackInfos.nb_ranged or source_village_army.nb_infantry < attackInfos.nb_infantry or source_village_army.nb_cavalry < attackInfos.nb_cavalry) { + return error.NotEnoughUnitsInTheVillage; + } + + var c1 = try db.savepoint("c1"); + // Else we remove the units from the village and create the attacking army + try c1.db.execDynamic("UPDATE armies SET nb_ranged = nb_ranged - ?,nb_cavalry = nb_cavalry - ?,nb_infantry = nb_infantry - ? WHERE id = ?;", .{}, .{ attackInfos.nb_ranged, attackInfos.nb_cavalry, attackInfos.nb_infantry, self.army_id }); + // And create new army + try c1.db.execDynamic("INSERT INTO armies (nb_ranged, nb_cavalry, nb_infantry, player_id) VALUES (?, ?, ?, ?);", .{}, .{ attackInfos.nb_ranged, attackInfos.nb_cavalry, attackInfos.nb_infantry, self.player_id }); + c1.commit(); + + const created_army: *Army = try allocator.create(Army); + created_army.* = .{ + .id = @intCast(c1.db.getLastInsertRowID()), + .nb_ranged = attackInfos.nb_ranged, + .nb_cavalry = attackInfos.nb_cavalry, + .nb_infantry = attackInfos.nb_infantry, + .player_id = self.player_id, + }; + return created_army; + } +#+end_src diff --git a/content/docs/project-1/entities.org b/content/docs/project-1/entities.org index 8da0d74..396d840 100644 --- a/content/docs/project-1/entities.org +++ b/content/docs/project-1/entities.org @@ -3,3 +3,95 @@ #+hugo_cascade_type: docs #+math: true +My video game consists of many entities, those have to be represetend, stored and easily manipulated through my Zig code. Since Zig does not support [[https://en.wikipedia.org/wiki/Object-oriented_programming][Object Oriented Programming]] I store all the data in =structs= and simple functions to manipulate my entities. + +Even though some of my entites are a bit more complex than others because they include things like inheritance, the basic entity in my project look something like this. + +#+begin_src zig + const std = @import("std"); + const sqlite = @import("sqlite"); + + // Make the whole file the Player struct + const Player = @This(); + + // Used to return to the user the ranking of the players without giving authentification informations + const Result = struct { + username: []const u8, + gold: i32, + }; + + // Fields of the Player struct + id: usize, + username: []const u8, + password: [32]u8, + session_id: [32]u8, + + // Function to update the state of the Player in the database, common to most entities + pub fn persist(self: *Player, db: *sqlite.Db) !void { + const query = + \\UPDATE player SET username = ?, password = ?, session_id = ? WHERE id = ? + ; + var stmt = try db.prepare(query); + defer stmt.deinit(); + + try stmt.exec(.{}, .{ + .username = self.username, + .password = self.password, + .session_id = self.session_id, + .id = self.id, + }); + } + + // Return a Player object fetched by the database + pub fn initPlayerBySessionId(db: *sqlite.Db, allocator: std.mem.Allocator, s_id: []const u8) !*Player { + const query = + \\SELECT id, username, password, session_id FROM player WHERE session_id = ? + ; + var stmt = try db.prepare(query); + defer stmt.deinit(); + + const row = try stmt.oneAlloc(Player, allocator, .{}, .{ .session_id = s_id }); + const player: *Player = try allocator.create(Player); + if (row) |r| { + player.* = r; + } else { + return error.PlayerNotFoundInDb; + } + + return player; + } + + // Same as above but with different parameter + pub fn initPlayerById(db: *sqlite.Db, allocator: std.mem.Allocator, id: usize) !*Player { + const query = + \\SELECT id, username, password, session_id FROM player WHERE id = ? + ; + var stmt = try db.prepare(query); + defer stmt.deinit(); + + const row = try stmt.oneAlloc(Player, allocator, .{}, .{ .id = id }); + const player: *Player = try allocator.create(Player); + if (row) |r| { + player.* = r; + } + + return player; + } + + // Function on the entity that is going to be called in files like game.zig where the basic logic of the game is, this is done to avoid having SQL queries in the main logic file of the server + pub fn ranking(db: *sqlite.Db, allocator: std.mem.Allocator) ![]Result { + const query = + \\ select username, villages.gold from player + \\ inner join villages on villages.player_id = player.id + \\ order by villages.gold desc; + ; + var stmt = try db.prepare(query); + defer stmt.deinit(); + + const players = try stmt.all(Result, allocator, .{}, .{}); + + return players; + } +#+end_src + +Even though OOP is not supported in Zig, I can still pretty easily map my database representations into my Zig code. In order to see more complex situations like inheritance, you can check the *Interfaces* section of this documentation. diff --git a/content/docs/project-1/events-handling.org b/content/docs/project-1/events-handling.org index 6fef09f..c64aeed 100644 --- a/content/docs/project-1/events-handling.org +++ b/content/docs/project-1/events-handling.org @@ -3,3 +3,75 @@ #+hugo_cascade_type: docs #+math: true +In my project there are a lot of =events= that happen when no player is connected or making any requests. Especially giving golds to the player each X ammount of time in order to give them the ammount of =gold= their =gold_mines= are producing, as well as treating the =events= like =battles= or =ressources_transfers= that are not resolved yet because of the travel time from one village to one other. + +So the server has to find a way to work even no request is submitted. That is where threads come in handy, I am going to spawn one thread for ressources polling and one thread for events polling. Those threads are not going to be constantly polling in order not to overload the database. + +So in my =main= function, before listening to the requests, I am going to spawn those two threads that are going to be running in the background. + +#+begin_src zig + // Start workers + _ = try std.Thread.spawn(.{}, ressourceProductionPolling, .{&sqldb}); + _ = try std.Thread.spawn(.{}, eventsPolling, .{&sqldb}); +#+end_src + +The ressource polling function is quite simple, it is going to update the ammount of =gold= the =players= have by directly hitting the DB. + +#+begin_src zig + fn ressourceProductionPolling(db: *sqlite.Db) !void { + while (true) { + //std.debug.print("Polling \n", .{}); + const start = std.time.milliTimestamp(); + + const query = + \\ UPDATE villages + \\ SET gold = gold + subquery.total_rod + \\ FROM ( + \\ SELECT villages.id AS village_id, SUM(productivity) AS total_rod + \\ FROM gold_mines + \\ INNER JOIN buildings ON building_id = buildings.id + \\ INNER JOIN villages ON villages.id = buildings.village_id + \\ GROUP BY villages.id + \\ ) AS subquery + \\ WHERE villages.id = subquery.village_id; + ; + var stmt = try db.prepareDynamic(query); + defer stmt.deinit(); + try stmt.exec(.{}, .{}); + + const end = std.time.milliTimestamp(); + const elapsed = end - start; + std.debug.print("Ressources polling took {d}ms\n", .{elapsed}); + + std.time.sleep(60 * std.time.ns_per_s); + } + } +#+end_src + +The thread polling for =events= is a bit more complex but you can get more details in the *Interfaces* section of this documentation. + +#+begin_src zig + fn eventsPolling(db: *sqlite.Db) !void { + while (true) { + const start = std.time.milliTimestamp(); + var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator); + defer arena.deinit(); + const allocator = arena.allocator(); + + const events = try Event.getAllTheNextEvents(db, allocator); + for (events) |event| { + if (try event.getRemainingTime() <= 0) { + try event.executeEvent(db, allocator); + } else { + break; // this event is not over so the next ones wont be either + } + } + const end = std.time.milliTimestamp(); + const elapsed = end - start; + std.debug.print("Events polling took {d}ms\n", .{elapsed}); + std.time.sleep(5 * std.time.ns_per_s); + } + } +#+end_src + +We can then see that even with a simple HTTP server we can still leverage the easy Zig std implementation of threads to make our server behave in real time without having to wait for HTTP requests. diff --git a/content/docs/project-1/shutdown.org b/content/docs/project-1/shutdown.org index 3d263b6..5736123 100644 --- a/content/docs/project-1/shutdown.org +++ b/content/docs/project-1/shutdown.org @@ -3,3 +3,52 @@ #+hugo_cascade_type: docs #+math: true +Since the function that starts the server and make it listen for incoming requests is blocking, I had to find a way to gracefully shutdown the server because I want to check for things like memory leaks, running time and everything else that might be necessary to gracefully shutdown the server. + +#+begin_src zig + defer iNeedThisToExecute; // will not be executed when the process is killed + try server.listen(); // Blocking call +#+end_src + +In order to do that I implemented [[https://faculty.cs.niu.edu/~hutchins/csci480/signals.htm][signals]], to catch the =SIGINT= signal that is sent when you press =CTRL+C= in the terminal. Because the default behavior is simply to kill the process. To do that I wrote a =sigaction= that is going to modify the behavior of the =SIGINT= signal. + +#+begin_src zig + var sa = std.posix.Sigaction{ + .handler = .{ + .handler = &signal_handler, + }, + .mask = std.posix.empty_sigset, + .flags = 0, + }; + try std.posix.sigaction(std.posix.SIG.INT, &sa, null); + + // Start server + start_time = std.time.timestamp(); + + try server.listen(); // Blocking call +#+end_src + +Now the default behavior of the =SIGINT= signal is going to be replaced by the =signal_handler= function. + +#+begin_src zig + // Working alternative to a defer in the main function + fn signal_handler(_: c_int) align(1) callconv(.C) void { + std.debug.print("Received SIGINT\n", .{}); + + server.stop(); + std.debug.print("Server stopped after {d} seconds\n", .{std.time.timestamp() - start_time}); + server.deinit(); + + // const deinit_status = gpa.deinit(); + // if (deinit_status == .leak) { + // std.debug.print("Memory leak detected\n", .{}); + // } else { + // std.debug.print("Memory freed correctly\n", .{}); + // } + std.process.exit(0); // important in order to kill all the pollings threads + } +#+end_src + +Note that here I commented all the memory leak detection because there currently are issues with the way the httpz framework handles the memory. I am currently in active discussions with the maintener of the framework to fix this issue. After that I should not have any memory leaks since most of the allocators I used are those provided by the framework. + +To silence the memory leaks and have better performances I use the =std.heap.page_allocator= for the moment. diff --git a/content/docs/project-1/tests.org b/content/docs/project-1/tests.org index 93f6aa7..bb52882 100644 --- a/content/docs/project-1/tests.org +++ b/content/docs/project-1/tests.org @@ -1,4 +1,4 @@ -#+title: Continus Integration (CI) +#+title: Continuous Integration (CI) #+weight: 4 #+hugo_cascade_type: docs #+math: true diff --git a/static/HEIG_ZIG/images/db_perf.png b/static/HEIG_ZIG/images/db_perf.png new file mode 100644 index 0000000..79bdad8 Binary files /dev/null and b/static/HEIG_ZIG/images/db_perf.png differ diff --git a/static/images/db_perf.png b/static/images/db_perf.png new file mode 100644 index 0000000..79bdad8 Binary files /dev/null and b/static/images/db_perf.png differ