Community Standup!

[adrianhall]

Hey Folks!

This will be a long running thread. Each Friday (unless I am on vacation), I'm going to be posting "what I did this week, and what is in plan for next week" - a community standup, if you like. Feel free to reply to the standup with questions, notes, observations - basically, whatever you want.

[adrianhall]

Community Standup: 12/3/2021

This week:

• Worked on the test re-organization for #261. The "live server" now uses an in-memory SQLite database rather than the InMemoryRepository, and lives in Datasync.Common.Test. In addition, there is a "integration test" project, which contains all the tests that go against the live server. What's left in the other test suites are unit tests that do not require a live server.
• Fixed #259 - the fix is in the QueryAsync method of the TableController, and happens in two places. In both cases, the fix is to try the SQL version, and if that errors out with an InvalidOperationException or NotSupportedException, then try the client-side version. If the client-side version errors out, then that results in a BadRequest or InternalServerError, depending on the error. If the client-side version is tried, then a warning message is logged. It might be ok for a one-off (like tests), but you definitely don't want client-side evaluations to be happening in your access control provider (for example).
• Created PR (#259, #261) Test Re-organization and client-side execution #272 for the code - it's currently broken because of an unreliable integration test, so I need to do multiple runs to find out where the stability issue is, fix it, and update the PR.

Next week:
I'm at the Azure + AI conference next week, so not a lot of work will be done. However, I'll try and get the test stabilization done. After that, I can release 5.0.0-beta.11

[adrianhall]

Community Standup: 12/10/2021

This week:

• Got the tests stabilized, so they work mostly. There is a DateTimeOffset comparison check that should have been done at the ms level (which is the level of granularity the SQLite driver handles) but was being done at the tick level. The rounding causes an intermittent error.
• Release 5.0.0-beta.11 with the two fixes plus the test reorganization.
• Started a feature branch for the necessary work on the sync client, which is the next big chunk of work to be done.

Next week:

• Continuing work on the sync client, with an in-memory local table provider (for testing purposes). I don't think I'll manage to finish it this coming week, but I'm hopeful to complete the work by Christmas.

Request for volunteers

Want to contribute? I'd like to put together a set of tutorial "Todo" apps across all the platforms we want to support:

• Xamarin.Forms (iOS, Android, UWP)
• Xamarin.iOS
• Xamarin.Android
• Xamarin.Mac
• UWP
• WPF
• Avalonia UI (iOS, Android, WPF, UWP)
• Uno Platform (iOS, Android, WPF, UWP)

Each one should be canonical (i.e. don't try to be fancy) and use the TodoApp.Data project for data access (allowing us to write common tutorials) if possible (if it isn't possible or you need to do something different because of the framework, then fine).

I'm going to quickly create the issues - just let folks know you are working on a specific bit in the issue, then I'll assign you and you can submit the PR when ready.

[ideas2appsaustralia]

Adrian
I have just got an App service with sql DB, .NET 6 asp.net core API, xamarin.forms MSL and B2C auth working for which I built all based on your eBook and ASP.Net core blog articles. The B2C stuff I got working with help from some of your comments and other articles and ALOT of grinding!

So Ive at least got experience creating things from YOUR docs and code base so I am available for any support you might want in terms of proof reading your tutorials , testing them as a user etc. Not at the point I could do anything on code for you though sorry

scott

[ideas2appsaustralia]

I will work my way through the wiki and convert over to Datasync from Azure Mobile Apps as was stilling using those formerly known packages
Scott

[adrianhall]

The official documentation is oh-so-close to being updated. I just need to line everything up for a GA release.

[adrianhall]

Community Standup 12/17/2021

This is the final standup of 2021. I'll probably work on this project over Christmas, but I won't be posting about progress until January. Enjoy the holiday!

This week:

• Release 5.0.0-beta.11 with the changes from last week.
• Updated the tutorials on docs.microsoft.com for authentication. Azure App Service rolled out a new UI for configuring authentication, so the old instructions were no longer valid. The changes were relatively straight forward, but I still had to walk through each tutorial to make sure they worked.
• Started work on #234. This is going to be an extensive change, so I expect it to be a month or more in the making. At the end, I will have an in-memory cache for offline usage. Hopefully, that will unblock anyone who wants to try their hand at the new offline experience.

Next two weeks:

• Continue working on #234. The first task is to start writing the offline tests, then I can start coding the actual implementations.

[myokeeh]

@adrianhall, regarding the second bullet, the docs I see that mention "authentication" are dated 07/26/2021. Am I looking at the wrong place?

https://docs.microsoft.com/en-us/azure/developer/mobile-apps/azure-mobile-apps/howto/authentication
https://docs.microsoft.com/en-us/azure/developer/mobile-apps/azure-mobile-apps/quickstarts/uwp/authentication

[adrianhall]

No - right place. The authentication setup in AAD is an include file, so it was updated. The top document (that includes the instructions) was not updated, so the date was not updated.

[adrianhall]

Community Standup 1/21/2022

Hey folks! Welcome back to the Community Standup for 2022! Apologies for the extended time off - vacations and getting back into the swing of things took some time.

This week:

• Figured out I was heading in the wrong direction with the offline work for #234. Took a look at the original code, and realized that I had done something rather stupid in leaving off the untyped table work when I did the new client. So I had to rewind a bunch of work. Now I'm heading in the right direction, but it was a lot of wasted effort.

Next week:

• More work on #234. Don't believe anyone who tells you offline is easy!

[adrianhall]

Community Standup 1/28/2022

Hey folks! It's been a busy week in the offline world. After switching the client back to Newtonsoft.Json, I implemented the untyped table, which can be accessed as table.GetRemoteTable("tableName"). All the methods produce JObjects, ready for the offline work. I rounded off the week by re-implementing about 50% of the integration tests against the untyped tables. I've done Get, Insert, Delete. Still to do: Query, Undelete, and Update. I'll complete those this weekend.

Next week, I'll be re-implementing the typed table table.GetRemoteTable<T>("tableName"), including all the integration and unit tests.

[adrianhall]

Community Standup 2/4/2022

I finally got a working offline-capable client working end-to-end! There are a bunch of changes and refactorings I need to do before I consider it stable, mostly because I dragged a bunch of code over from the old client to the new client to deal properly with the JSON based operations queue. Things left to do:

• Clean-up after I switched to string IDs and removed CreatedAt as a required field.
• Consolidate all the JSON parsing into one place (it's all over the place in the old client and enough code came over that it needs clean up)
• Moving the store operation over to a struct so that the operations queue is more efficient.
• Switching the threading utilities out for the latest .NET versions.
• Doing better thread pooling on the HttpClient to ensure that we can do multiple pulls at the same time.
• Implementing cancellation tokens everywhere
• Switching to System.Text.Json (by a quirk of design, System.Text.Json documents are immutable and several places in the operational queue like to modify the document; if you switch to a JsonNode, then you get mutability but lose other things which are good)
• Implement the client options for offline.
• Check out how the authentication provider works with offline (it should just work since it uses the same underlying HTTP transport, but better to be sure)
• Probably other stuff that I haven't discovered yet but will no doubt rear their ugly head as I am refactoring.

So, I think the term "progress was made this week" is appropriate.

[adrianhall]

Community Stand up 2/11/2022

This week, I have been working on getting all the tests working; most notably, the old client had a whole slew of integration tests that were run on devices. I'm not there yet, but I'm going to be running the tests on a.NET STANDARD setup. I'm about 25% done - the remote table using JSON objects is mostly, to be precise. I added some code around returning search results as IAsyncEnumerable, so that requires net new testing which hasn't been completed yet.

Next week, I am going to finish the remote table testing and move onto the operations queue testing.

[mjfusa]

@adrianhall Is there a roadmap for major releases post-beta? (Preview, GA) If not can you share a rough timeline? Thanks!

[adrianhall]

I won't be posting a roadmap until I've done a beta release of the offline store. Once that is done and tested, I'll have a better idea of the timeline.

[adrianhall]

Community Stand up 2/18/2022

This week was a busy week, but with little progress. First off, when I created the tests for the LINQ expressions, I discovered a bug in the way that filters are handled. It took me a solid 3 days to track down that bug 🤬 It turned out to be a one character bug (!= instead of ==), but that was enough to set me back 3 days. After that, I had to update our CI/CD pipelines. At Microsoft, we are serious about secure supply chains, ensuring the NuGet package that you download is what we intend and doesn't have any malicious code in it. To ensure that, I have to use internal build machines and have a bunch of additional steps behind the scenes when I release code.

After that, I switched back to generating tests for the offline work; and there is a lot of it. I checked in all the tests for the remote version of the tables (and fixed some nagging bugs at the same time). Currently (and for the next few weeks), I'll be working on the operations queue tests primarily, and hopefully checking in the SQLiteOfflineStore package as well.

Until next week!

[adrianhall]

Community Stand up 2/25/2022

Hey folks. This week was more test writing. You are probably all wondering when the next release will come out and what it is gated on. Well, the old client did not have any tests for the underpinning code, relying on integration tests with an actual server to do the testing. However, I found many instances when the code was "wrong" but the integration worked purely by accident. For example, take a look at this code:

        private static bool IsResponseCompressed(HttpResponseMessage response)
        {
            response.Headers.TryGetValues("Content-Encoding", out IEnumerable<string> EncodingList);
            if (EncodingList == null)
            {
                response.Headers.TryGetValues("Vary", out IEnumerable<string> VaryList);
                if (VaryList == null)
                {
                    return false;
                }
                string allVaryValues = VaryList.Aggregate((allValues, next) => allValues = allValues + ";" + next);
                return !string.IsNullOrEmpty(allVaryValues) && allVaryValues.Contains("Accept-Encoding");
            }

            string allAcceptEncodingValues = EncodingList.Aggregate((allValues, next) => allValues = allValues + ";" + next);
            return !string.IsNullOrEmpty(allAcceptEncodingValues) &&
                 (allAcceptEncodingValues.Contains("gzip") ||
                 allAcceptEncodingValues.Contains("deflate") ||
                 allAcceptEncodingValues.Contains("br") ||
                 allAcceptEncodingValues.Contains("compress"));
        }

There were no tests for this piece of code. However, there are at least 3 distinct bugs, one of which I could see through inspection pretty easily. (I will leave the identification of the bugs to the reader - can you spot them all?) Refactoring this as an internal extension method allows me to test this in isolation, which resulted in better code and more test coverage. However, the three bugs ate up two days of time. A little bit of due diligence in generating tests for code that is deep in the library allows me to be more confident later on when I start adding features.

So, the next few weeks are going to be relatively tedious - writing tests for places where there is no test coverage, understanding what the code is meant to be doing, and then refactoring the code to work better for the new client.

[adrianhall]

Community Standup 3/4/2022

Hey folks - getting closer now to an offline release. I've been refactoring a lot of the pull and purge operations so that they work with IAsyncEnumerable<T> operations. I also found that operation cancellation didn't go all the way down to the bottom, which would have left threads in a problematic state with no way to find out the problem thread; so I fixed that - all async operations (and they are all async operations) now have full top-to-bottom cancellation.

My test crusade is also continuing. When we do an offline query, we parse the OData query that we would normally send to the server into a QueryDescription which can then be re-formatted as an SQL statement. Unfortunately, the parser had a few bugs in it; ones you wouldn't bump into in "normal" usage, but if you were doing complex pulls based on the content of records, you would definitely see it. The tests now cover more than 90% of the cases for that parser, so we should be good in the future.

Finally, I did away with the queryId. Incremental sync is now the "normal" way of doing things - I create a queryId for you from a hash calculation of the query plus table name, then store that as the delta token key.

This next week has me finishing the deep dive on the pull code, and then working on the tests some more. Given the switch to IAsyncEnumerable<T>, I'm hoping I can switch out most of the code for a simple loop - we'll see how that goes.

For those wondering, we are probably 2-3 weeks away from the next beta.

[adrianhall]

Community Standup 3/11/2022

This has been a really slow week filled with things not related to coding, so not much progress has been made this week. I did get more code coverage in the Query module and over the SyncContext, as well as uncovering another couple of bugs in the original code that I corrected.

One of the things I have not covered yet is eventing. The old client had a rather complex system - you created an EventManager and then selected the events you wanted from the set of possible events. You could then subscribe to the event stream via the EvetnManager. I'm thinking of switching this to a standard .NET event handler - there would be an event handler on the client and one on the store. So, for remote events, you could use:

client.RemoteEvents += (sender, args) => { /* do something here */ }
store.OfflineEvents += (sender, args) => { /* do something here */ }

Of course, the event handlers are all synchronous code. So the alternative is to have some sort of register:

var disposable = client.RegisterEventHandler(IEventHandler eventHandler);
/* eventHandler.OnEventReceivedAsync(event) */

The disposable would unregister the event handler when disposed. Either way, they would run on the currently active thread, so you would have a bunch of RunOnUIThread(() => ...) type calls to effect changes in your mobile UI.

Do you prefer async or synchronous event handlers?

[adrianhall]

Community Standup 3/18/2021

I had a lot of "non-OSS" stuff going on at work this week, so definitely not as much progress as I wanted. What I've been doing is going through all the code to see which code paths are actually tested and ensuring that there is at least a test covering them - where things go wrong, they should throw an exception, or return null, or whatever the appropriate case is. This ensures that, for instance, if you give the library some bad Data query, we know it will throw appropriately and not take the rest of the stack down with it.

I also took a first stab at the in-memory store. This turned out to be a lot more complex than I expected. It's easy (relatively) to turn LINQ queries into something string based, but a lot harder to turn the Data query back into a filter. I've shelved this part for right now, and I'm going back to my main focus.

Finally, I got the PurgeItemsAsync() method within the SyncContext working and fully tested. I'll be moving on to the PushItemsAsync() method next. One of the things I'm quite keen on is to ensure that each table is fully separated - if you want to only push one table and then pull that table, you should be able to do so. It's the pre-work necessary to do multi-threaded pushes, which will speed up the time taken to do a push considerably.

[fileman]

Pull, Push and Purge for single table is a great improvement respect old client.

Why you havent created a public branch?

[adrianhall]

The code is not directly useable yet - it's only useable by the unit tests. One of the last things to be done is the wire up to the OfflineTable object, which also adds specific tests in for that functionality. Once I've done that wiring (and before I've done the integration tests), I'll push to main for everyone to see.

[adrianhall]

Community Standup 3/25/2022

I've finally done the Pull Request that integrates the majority of the offline synchronization functionality. It's a big PR, and I didn't even do everything I wanted to do! Given my recent focus on testing, you can guess that what is left is a big bunch of unit tests and the integration tests (which will have to wait). I've done all the main logic, but I need to write unit tests for the actual offline table (which should be easy since the underlying sync logic is already unit tested), conflict handling and OData to LINQ parsing. I should have that done by end of next week.

Still to do before I can make another preview release:

• OfflineTable<T> generic offline tables.
• The SQLiteOfflineStore.

[adrianhall]

Community Standup 4/1/2022

This week:

• Completed all the work for data synchronization, including all the unit tests. This closed #234.
• Completed the major work around implementing the SQLiteOfflineStore for #235.

What's left before the next beta?

• Unit tests for #235.
• Integration tests for offline tables.

Interesting notes on the new implementation:

1. You can now use ExecuteQueryAsync with a definition of the result set to do joins that are properly serialized. This is the first step in working out one of the annoyances for me - doing multi-table joins in the generic (strongly typed) world.
2. Instead of a plain filename, you provide either a full qualified filename (starting with '/') OR a URI. The URI is nice because you can specify in-memory stores.
3. The old version had to code CEILING() and FLOOR() due to limitations in the older SQLite implementations. The new ones have those functions, so I just used them. Similarly, the max parameters per query was increased from 999 to 32K, which I took advantage of for faster bulk updates.

Additional things I want to do:

1. Have a constructor that takes an sqlite3 or a SqliteConnection. The former would allow you to set up the store based on any SQLitePCLRaw based database connection that you have set up. The latter is a helper method that just deals with setting up the SQLitePCLRaw database and preparing statements. Both mechanisms would allow you to more easily handle encrypted SQLite databases.
2. Have a generic ExecuteQueryAsync<T> which returns an IEnumerable<T> - should make it easier to do complex queries.

I won't be doing the extra stuff until I've got the next beta out. I am aiming for getting the unit tests written by the end of next week. If I can motor through those, I may even get the integration tests done this coming week as well.

[adrianhall]

Hey folks - an early morning (for me) Community Standup 4/8/2022

Why so early? Because I have just dropped v5.0.0-beta.12 onto NuGet. This includes the updated (well, more like older) client and SQLite store, updated for the latest Android and iOS releases. There are a couple of changes in place:

• You can push, purge, and pull individual tables.
• Date/Times are stored in the SQLite store with ms accuracy.

Since the SQLiteStore NuGet is new, it will take a while to hit the indexes. The others are available immediately. There is no change in the server-side, but significant changes in the client side. Feel free to ask questions in the release discussion and I'll try to keep ahead of them.

My next focus is to work on the samples, tutorials, and documentation. That requires a shift to my Mac, which I haven't updated in some time. So nothing much will be happening this weekend, but I hope to get to some updates to the samples this week. Finally, I've started doing some thinking and research on what a .NET 6 MAUI client would look like. Since the future for mobile development at Microsoft is all .NET 6 MAUI, I'll be focusing on that platform going forward.

[adrianhall]

** Community Standup 7/1/2022 **

Hey folks. It's a long weekend here in the US for the Independence Day national holiday, so I'm looking forward to some sun, bbq, and fireworks this weekend. I hope my US based devs will likewise take a break.

This week, I did a bunch of things:

• There is a new MappedTableController which uses Automapper to translate between DTO and Entity. It wraps another repository, so you would use it with EntityTableRepository (for example).
• I've implemented the deferred table creation. You no longer need to "DefineTable" if you are using generics - just pull the table from the server. i.e.

var store = new OfflineSQLiteStore(connectionString);
var options = new DatasyncClientOptions { OfflineStore = store };
var client = new DatasyncClient(endpoint, options);

then later:

var table = client.GetOfflineTable<Model>();
await table.PullItemsAsync();

No define / initialize is required. This only works properly if you GetOfflineTable matches the table definition and you are using the generic version. If you are using the JSON version OR the table definition you are using differs from the one in the backing SQLite database, then you have to do the old method.

I also fixed a few bugs, increased test coverage, and worked on documentation. The two new mechanisms are documented in the wiki, but they won't make the main docs until I release 5.0.7 (which is likely to be next week).

This week, I'm working on NSwag support. This will be another new package that allows you to integrate an OpenAPI 3.x swagger UI. Neither NSwag nor Swashbuckle work with generic controllers, filters, or OData. We have to create an appropriate operation processor to add the capabilities in so that the right document is output. I'm also going to update the template. It was written when EF Core was still an RC. Now it's been released, the dependency packages need to be updated.

I'm expecting to release 5.0.7 next Friday, so stay tuned for that!

[adrianhall]

Happy weekend everyone!

This week, I'm releasing twice - v5.0.7 just hit NuGet with the MappedTableController, ability to set the HttpTimeout, an update to the template to move EF Core to the GA libraries (thanks @fileman) and a few other minor bug fixes. I'm also in the process of releasing v5.0.8, which corrects an issue caused by the iOS HttpClient in MAUI. Basically, there is a possibility of duplicate headers appearing in both HttpResponseMessage.Headers and HttpContent.Headers. When we merge the headers, this causes a duplicate key error. Curiously, it only affects the iOS MAUI HttpClient and not other platforms. Anyway, if you are attempting to write MAUI apps, you will want to upgrade to v5.0.8; otherwise you can skip it.

Today, I'm working on the MAUI tutorial on Mac (including the iOS tutorial - guess where I found the bug!) and then I'm going to be working on the Uno Platform sample. That will close off the .NET client for a while as I will be switching my attention to other client libraries - most notably the TypeScript library, supporting React Native first (with a SQLite store) and then browser support (with an IndexDb store) which will allow you to write offline PWAs. I'll be looking for volunteers to assist with the work, so if you happen to be a JavaScript dev and want to contribute to the JS client library, let me know!

[adrianhall]

... and I'm an idiot who didn't set up NuGet publishing for the new Automapper library, so I'll be releasing a new v5.0.9 that will hopefully correct that!

[adrianhall]

Community Standup 7/15/2022

This week saw the start of the TypeScript client. You can find the work happening in my local branch if you want to follow along. Thus far, I've got all the build/test/api report/code coverage done., plus a handful of validators needs across the client and the base extensible HTTP client done (based on the axios library), and the tests for everything done thus far. It's not runnable as a client yet, but I expect to make rapid progress towards building the remote client and releasing a preview. Ramping back up on TypeScript after the better part of 9 months coding exclusively in .NET took longer than expected, so I didn't get to the Uno Platform tutorial (if anyone wants to write the base sample, I'd appreciate it).

[adrianhall]

Community Standup 7/22/2022

This week, I've continued working on the TypeScript client. The work on the HTTP client portion is mostly complete (I'm using both Fetch and Axios as examples), and I'm working on tests for the rest of the day to close out the week. Next week, I am going to start on the query library (providing the capability to run something like LINQ in TypeScript). So progress is being made on the TypeScript library.

[adrianhall]

Community Standup 7/31/2022

This week has been brutally hot in Seattle. We're not prepared for extreme heat (no air conditioning), so when it happens I hibernate in the basement rather than working on code. So, not as much happening as you would expect this week. We do have a couple of outstanding bugs in the .NET client - good news is that I know how to fix them. Bad news is that I haven't gotten to it yet. That's mostly because I've been hibernating. I've also been working on the TypeScript client - the HTTP module is mostly done; all the serialization and deserialization is checked, and I started working on some tests for the LINQ part of it. The good news here is that the Azure SDK folks have made a really solid HTTP pipeline and paging library (check out @azure/core-client and @azure/core-paging for more details) - these are both cross-platform (browser, electron, and node) out of the box, which makes the process easy.

Next week, I hope it will cool down enough that I can finally get to the .NET bugs. I'm going to aim to release a fixed version at the end of next week. Then I'll switch back to the TypeScript client and finish off the HTTP client implementation.

[adrianhall]

Community Standup 8/5/2022

Happy Friday everyone! A bunch of updates this week.

• On the .NET front, I fixed a bug reported by @wedgeley for ordering of synchronized items to ensure the delta token store is updated properly. This should also another issue that was reported - so we will see.
• Also on the .NET front, there is now a method on the offline store to get a list of tables - thanks for the suggestion @henda79 !
• On the TypeScript side, the cross-platform HTTP client and serializer is done, including full tests - I'm moving onto the table definitions right now.

One more bug was reported - the Dispose() method of the offline store is not propagated, meaning that the database file is kept locked. I'll work on this over the next couple of weeks.

I should be releasing .NET client 5.0.10 this afternoon (assuming the pipelines don't throw any errors). So watch for that next week! If you are using the offline functionality, I'd recommend updating to the latest version.

[adrianhall]

Community Standup for Aug 12, 2022.

Hey folks, another week gone. I had to revamp the TypeScript client (I’m using @azure/core-client for the main functionality - this is the same cross-platform client that all the Azure SDKs use). There were a couple of things that didn’t quite work, and a quick chat with some engineers in the organization made me realize that I was doing too much abstraction, so everything has been dialed back, resulting in a much leaner codebase. I’ve got most of the code written for remote tables, and I’m working on design problems associated with the offline stores. I want to provide a SQLite option for mobile applications (like React Native) and a browser based offline store as well. It’s turning into a bigger design problem than I expected.

On the .NET side, 5.0.10 was released. I do have one pending change in the main branch that isn’t in the released branch - there is no Dispose() method to close the offline store, which affects desktop only (since force quit on mobile closes the connection). I’ll release the 5.0.11 next Friday with this fix in it.

Finally, I’m going to be taking some long overdue vacation in September, so don’t expect updates or code in the first two weeks of September.

Until next week!

[adrianhall]

Community Standup Aug 19, 2022

Happy Friday, everyone. We've got a new release to talk about! v5.0.11 contains support for Cosmos Db. How do you use Cosmos Db? First, you need to go read all about the Entity Framework Core support for Cosmos Db. Now, let's discuss how you are going to make it happen:

1. Make your model derive from ETagEntityTableData instead of EntityTableData.

2. Add an OnModelCreating() method so that you set model.EntityTag to have the ETagConcurrency flag:

   protected override void OnModelCreating(ModelBuilder builder)
   {
       builder.Entity<TodoItem>().Property(t => t.EntityTag).IsETagConcurrency();
       base.OnModelCreating(builder);
   }

You can do anything else in that the provider docs tell you (see link above), including setting the partition key and container for the table. You just need to make sure that the EntityTag is wired up properly.

v5.0.11 will be coming out this afternoon (pacific time - it's currently being built and tested), and that will quickly be followed by updates to the documentation and a new Cosmos Db sample by the end of the day. Thanks to @timClyburn for his testing of Cosmos Db as I hadn't realized it wasn't working.

On the TypeScript front, nothing happened - it was a busy week and I concentrated on the Cosmos Db support. Hopfully, I'll get back to TypeScript this week.

[adrianhall]

Community Standup Aug 26 2022

Happy Friday! This week in Azure Mobile Apps land, we have a first check-in of the TypeScript library. It is not published yet (and won't be for a while - there is still much work to be done). Right now, there is no offline work done. However, it does implement a repository pattern for the tables plus it has authentication, logging, etc. built in. It uses the @azure/core-client stuff underneath, which is the same set of libraries used by all the Azure SDKs. My aim is to try and get this published under the @microsoft organization, and I have to go through some red tape for that. Next up, I want to write the React / Angular / Svelte / Vue / React Native TodoApp using the existing service definition and get those published, so this week will be a lot of demo writing.

This is also the last Community Standup for the next four weeks. I'm going on vacation and will not be bringing the laptop with me, so there won't be any updates. The next update will be September 24th, so I will see you then!

[adrianhall]

Community Standup Sep 23 2022

Hey folks! Welcome back. After a much needed vacation (touring the tombs and temples in Egypt), I got right back at it. There were five issues I was targeting for this week, and the code has been checked in already.

1. (#470) If you didn't provide an expiry date when using the generic authentication provider, it would throw a System.ArgumentOutOfRangeException. Arguably, that's bad - but you should be providing an expiry date for all tokens because we can then cache them rather than calling your code to refresh them. However, in this case, if we get an exception when calculating if the token has expired, we just assume it is expired.

2. (#468) When using the AccessControlProvider, we were calling IsAuthorizedAsync() with the wrong table operation value. This has been fixed.

3. (#463) There is now an event delegate for reporting synchronization progress. You can use it like this:

   client.SynchronizationProgress += (sender, args) => { /* Your processing here */ }

   The args is a new type SynchronizationEventArgs - you will receive a start and end event plus pre- and post- item processing events. This can be used for progress indicators, activity indicators, or whatever else you want.

4. (#461) DatasyncClient now respects the path in the Uri provided. Before, if you provided something like https://foo.com/bar and you then accessed a table t1, it would look at https://foo.com/tables/t1. It will now look at https://foo.com/bar/tables/t1. In addition, the table endpoint resolver can return either absolute paths (/tables/t1) or relative paths (tables/t1) and they are treated appropriately. Absolute paths ignore the path of the provided Uri, whereas relative paths are appended to the path of the provided uri (as expected).

5. (#437) Previously, you could define a model with a non-string ID and it would be allowed until the point at which it failed - usually when changing from remote to offline usage. Since this was painful, I've now disallowed it and ensure that the model has a correct ID field type before using it.

That completes the work for 5.0.12 of the .NET client and server. There isn't anything that screams "Release now!" (except maybe #468 on the server side). As a result, I'm going to leave this release until next Friday. If you would like to download and check out the fixes, I'm happy to take bug reports and will fix them all the way up to the release.

[adrianhall]

Community Standup 9/30/2022

Hey folks - welcome to another community standup. I released v5.0.12 of the .NET libraries, which I mentioned last week. Just one additional feature got added between last week and this week - #484 - access to pendingoperations. I've also updated the documentation site with all the new features, so it's ready for use.

This week, I'm going to return to the tutorials - specifically, iOS .NET MAUI tutorial, which was giving me issues (unrelated to the Datasync library) previously. Hopefully, I'll be able to walk through that this week. I'll also be writing an Uno Platform tutorial! Once those are done, I'll be switching back to the TypeScript library.

Until next time!

[adrianhall]

Community Standup 10/7/2022

Short version this week, as I have had COVID since Sunday. Nothing has been done. Hopefully, I'll get to something this week, but I'm still not over the illness, so I'm not counting on it.

[fileman]

get well soon

[myokeeh]

Yes, get well soon!

[adrianhall]

Community Standup 10/14/2022

Another week of COVID and very little work being done. I did manage to get to the bottom of Visual Studio for Mac and MAUI. The short version is that there is a combination of stuff that needs to be at the right version, which is preventing me from running the MAUI sample on my M1 Mac Mini. It will get sorted out in the next release of .NET (.NET 7.0 RC2), so I'm in a holding pattern on that. I'll then have to loop back to the Windows version and update the instructions to ensure the instructions hold.

I've also started working on the Apache Cordova replacement tutorial. This is actually a smaller entity, since it doesn't have offline support right now. I've got a design for the TypeScript offline support figured out, but I don't have the code written yet. So things are progressing there as well.

I'm looking for volunteers to help with the writing of suitable samples. I need samples for:

• Project Uno (using the existing stuff in TodoApp.Data project)
• Web Frameworks (React, Svelte, Angular, Vue - again, with a common data project that can be shared between them)
• React Native

If anyone wants to volunteer for writing the first versions of these, let me know so we can discuss the PR, form factor, etc.

[adrianhall]

Community Standup 10/21/2022

Hey folks. This week was all about clean-up. I got a report of an issue with the WinUI tutorial. The problem wasn't an issue with the tutorial, but I did notice some things that were wrong - most notably, the Min version was set to 17763, but the new MSAL library required 19041, so I updated the min version. There were also a bunch of doc requirements to be done.

I got a base template for a React TodoApp that I can use to do a TypeScript demo. The defining feature, as with the .NET ones, is that I have a service component to isolate the changes needed during the tutorial. My next step is to actually do the tutorial and running locally. Once I get through the base version and the authentication version, I can start looking at the offline and LINQ capabilities within the typescript library.

Finally, I'm working through the necessary work needed to publish the TypeScript library once it's ready. There is a lot of red tape for this so that I can launch as @microsoft/datasync-client instead of ms-datasync-client.

I'm still looking for volunteers to help with the project - particularly writing decent samples. If anyone wants to work on this open source project, please let me know.

[adrianhall]

Community Standup 10/28/2022

Welcome to the last update of this sequence. There are a couple of reasons for this. Firstly, my day job is taking up more of my time, resulting in less time to spend on this project. The progress will inevitably be slower. Secondly, it's coming to the holiday season, and with that there is an additional slow-down. Finally, this thread is getting too long.

So, what's been happening. First off, I've been working on the React TodoApp for a TypeScript demo. That has allowed me to work on the authentication and offline capabilities. The library is not yet ready to launch, but I'm getting there. Secondly, I've done some updates to the documentation.

Finally, I've been giving some thought to an alternative sync strategy. One of the big problems with the library has always been the extra baggage that we've had to carry around with the models. We have to have a specific ID form, a version, and an updatedAt field that has millisecond accuracy. This is because when the first version was released, it was common to have smaller amounts of storage on mobile devices. We can, however, have less requirements by having a shadow copy of the data.

Consider if we had three copies of data - one on the server in the server database, one on the device that is a precise copy of the data pulled from the server database, and one that is the copy modified on the client. When the user modifies the data on the client, a copy of the server version is made on the device and the modifications are done there, including updating the "last updated" field.

Now, when we want to synchronize, we take all the updated versions, diff them against the server versions, and send the updates to the server. Then we pull the server image down (using incremental sync if the updatedAt field is present) and update our internal accounting.

This has a bunch of advantages but one big disadvantage. The advantages are:

• The only field we need is an updatedAt field for incremental synchronization. All other fields are optional and user generated.
• We can do patch updates, which will result in less conflicts.
• We can do batching and transactions are easier.

The big disadvantage is that we need two copies of the data - one from the server and one with the changes in it. I believe we can update this by providing a JOIN on device between the shadow copy (what is on the server) and the client copy (the updates that the client has done).

Hopefully, I'll be able to flesh out the design and work on a new library with the new philosophy and see what people think.