Full Offline Mode now in Calendar 1.0 rc2

Calendar 1.0 rc 2 now comes with offline support packaged for CALDAV and ICS calendar providers. For CALDAV calendar it allows retrying of events if for any reason, the updates failed to be sent to the server.


You can find it at its new location on the addons.mozilla.org beta channel (If you still get 1.0rc1, then please try again in a few hours when the page cache has been updated).


If you encounter any issues, please file a bug at bugzilla.mozilla.org.


Or you can share your experiences via comments below.


Happy to hear from you regarding your experiences. Hope you enjoy the feature.


Cheers

mohit

Pencils Down GSoC

With in a few hours the awesome  programme that kept me occupied in the summers will be going off. But its the spirit that shall be living will continue on and great things will happen along the way, I am sure of it.

So here's the status of the bugs I have been working on.

1. bug 380060 - Implementation of offline mode - Done !! some minor tweaking here and there is left. Code-review is in progress and would be completed soon.
2. bug 502936 - prevention of calendar refresh mode - Done !! review+ on bugzilla
3. bug 586276 - Addition of Hooks in lightning - This project was last year's GSoC for Lightning, the code was reviewed by Philipp and he would after some consultation with Wolfgang and Ludovic would move it to the trunk. So I can mark this as done as well.

So essentially all the bugs assigned have been dealt as solved. I am very happy to have contributed to such a large project taken care of by peeps and geeks in countries all around the world.

Truly amazing.

Reflections on the programme coming soon.

Cheers

MSK

Stepping into the end of August

I am coming close to the end of Google Summer of Code and its been really an enriching experience for me, both as a developer and as a student. The code is already in review stages and it will take some time before it is in the trunk and finally in the product.

These days I am busy improving the code and getting feedback from a lot of people, especially Philipp. He's been busy with exams and still contributes fulltime to mozilla. Amazing. Seriously.

Anyways I have looking up bugzilla to find some bugs in the mozilla calendar arena which i can contribute to, but i cant seem to find many which I can help with. All of them require a deep deep understanding of the entire system which I unfortunately do not have. But yeah, maybe sometime in the near future I can start contributing as a much better open source developer.

So some more bug hunting going on these days. :)

Cheers
Mohit

Fixing bugs and biting burgers

Hi all,

I am back in Singapore and its amazing how fast and efficiently Phillip Kewisch has been working all this while. I don't think he even thinks of holidaying in Germany. While I was travelling and settling into my new hostel in Singapore, he efficiently reviewed my offline-syncing code for GSoC 2011 and provided me with a number of feedbacks.

Among them the most important is to allow the user to distinguish whether the remote calendar is working at all. This is because under the current system, if the remote calendar is not working then the calendar entry will be pushed into the cache and will appear in the views by default. This is something I will have to discuss with Wolfgang and Ludovic. Apart from that a few styling issues and the patch should be ready to be applied onto the trunk.

The discussion is available here: https://bugzilla.mozilla.org/show_bug.cgi?id=380060. Share testing issues and your experience with the patch.

And I just finished porting the refresh killer bug 502936 to the Lightning 1.0 trunk. It should be available in the 1.0 release. Its high time Lightning stopped having beta releases all the time.

And now I working to finish off last year's GSoC project to allow for stubs in Lightning for extensions.

Long nights ahead.

Cherios
MSK

Patch ported and posted!

I just posted my first patch. Phew, it has been days of reconciliations between the inverse trunk and the lightning trunk, with a number of small differences to be resolved.

But, in the end, its been a positive weekend and week so far, I managed to clone the repository once from comm-central. Spent 4 hours in trying to get all the changesets, Mozilla source has really a long long history. Started a build environment once again and got the binaries i wanted.

So the patch has been tested on TB version> 5.0 and it works nicely in the offline mode as well as the 50x server unavailability mode.

Feel free to check out the patch. Its available here https://bugzilla.mozilla.org/show_bug.cgi?id=380060

Comments will be greatly appreciated. I am looking forward to some feedback to improve the user experience in dealing with these scenarios.

Cheers
Mohit

Porting the patch to trunk

ah, first post in a long time.

So I am in India, enjoying an awesome summer with my friends and family. Eating a lot and having loads of fun time. However, I am happy to tell you that the offline patch is now complete and just needs to be ported over to the Head branch of Lightning, once that is done, future versions of lightning will then have the ability to do all the funny offline stuff they ever wanted.

Isn't it cool?

And apart from that GSoC has reached its mid-point an evaluations are beginning to happen. More coming later.

Cheers
Mohit

New Feature Implemented: Handling of Server Errors

Hola!

When we implemented the offline sync mechanism it occurred to us that the offline mechanism could be extended to a large number of situations and one of them is the situation where one can handle the server availability errors to a large extent.

Caldav Calendar code has been modified to allow for addition, modification and deletion of events during the case where a server is unavailable and a status message of 50x is received on the remote calendar request by lightning.

We assume that a human can't do anything about a server error, and we also assume that the server will be fixed. So this does not mean that the events u manipulate will sync remotely. They will however, sync, whenever the server comes back online.

you can find the binaries here and again these are purely for testing purposes and any bugs you can share with me via comments or mail me at: mohit.kanwal(AT)gmail.com

cheers
msk

Binaries are online *Solely for getting Test feedback*

Hi everyone,

Binaries for the different platforms are online now at https://github.com/creativepsyco/Lightning-1.0-b2-Offline-Sync-Binaries . If you have an interest in helping me test out the Offline Sync feature you can go ahead, download the xpi file appropriate to your system and try out thunderbird with the offline sync enabled lightning. Or if you are just curious what this version of lightning can do create a dev profile in Thunderbird and install the add-on into that profile, help on this is available here: https://developer.mozilla.org/en/setting_up_extension_development_environment#Development_profile

I must stress that things may break, and when they do break, send me an email at mohit.kanwal(AT)gmail.com or  file an issue in github, I will then get in touch with you regarding the cause of the issue and sort it out.

So yeah, go ahead and give it a try. You would need a stable build of Thunderbird to use with the binary...

Looking forward to some feedback :-)

cheers
msk

And We are Getting There

Life's been pretty busy these days. Packing, Packing and only packing. Work on the offline feature is now edging closer and closer to testing phase. Bugs are being discovered everyday and fixed on a regular basis now.

The current state of the project allows a user to :-

• Edit, Delete and Create events offline and sync them back online
• Modify recurring events offline and sync them back online
• Handlers to detect cases when syncing is not possible if network connection is severed.
• Syncing the calendars on the loading
• Same handling for the Todos as well

The next set of items I am working on are:-

• Create a switching code that detects if additions, deletions etc are failing so that it can put the calendar in offline mode 
• Reconcile these items whenever calendar can be reached.

The above 2 are not even present in Apple iCal so this gives me a feeling that I am doing something new. I have not looked inside the new Lion OSX so I can't say that they have implemented this already, but lets see how this feature works. 

PS: If anyone is interested in testing the offline syncing features, then drop me a comment and i will host the xpi files somewhere. Or if you guys have any ideas that you wish to see in the feature let me know. Your feedback would be really really valued and appreciated, and plus I wish to stress that many many things might break if you are using a test version of Offline sync, so please don't use it for the real events.

Cheers

msk

Step 1: Doing offline changes to cached Events

This week a big step has been implemented and tested fully, the ability for the user to create and edit events in the offline mode. Such a step is significant since it leaves me with only to implement the sync algorithm. Wolfgang made several changes to the idl files and interfaces. The changes include:-

• calICalendar now has flags for OFFLINE_DELETED_ITEMS and created and modified items as well which allow us to changes the behavior of getItems() accordingly.
• There is a new interface calIOfflineStorage which defines the methods used to handle offline operations, currently only 3 methods are defined, offlineDelete, offlineAdd and offlineModify and calStorageCalendar.js which implements the backend for calIStorage interface, the basic SQLite handler for cached calendar, implements these methods. 
• The above methods changes the offline_journal flag, which is an additional field added to the SQL table cal_events and cal_todos. As such there was a need to write an upgrade function. This made me add code inside the calStorageUpgrade.jsm module, raising the database schema version to 20 from its current level of 19. 
• Since offline_journal is not a part of the standard calIEvent interface, there would always be hacks and workarounds all over the place since which writing and reading events, the functions inside calEvent.js and calTodos.js would destroy the sql columns of the record. For example, the modifyItem function inside calStorageCalendar.js first destroys the item in the database, then adds the new modified item. This a hack, wolfing told me and hence I must make sure that it does not screw up the offline_journal flag, hence before it actually goes ahead and deletes the item record, I get the flag and restore it when the modification has completed.

Coding wise, it all looks good and the wiki has been a life saver. Seriously without it there would be no progress at all. The next step is to get the sync in place. Steps for doing this have been outlined at http://wiki.mozilla.org/Lightning_Offline_Sync

cheers

msk

A New Turnaround

Yay, the project is going full steam now.

Wolfgang from Inverse and a contributor on a number of bugs in the calendar project has joined and is now directly overseeing the work I do.

I had a meeting with him and Ludovic and finalised the roadmap of the feature completion. The deadline we have set is to get the feature up and running in 3 weeks, which I think would not be difficult at all. And plus wolfgang is really helpful in speeding me into writing code and committing changes.

Wolfgang has created a branch at mtn.inverse.ca and I am working on getting the methods written down, while he writes the glue that is required to get the thing running smoothly.

Awesome Days ahead.
Mohit

Sequence of Events

A slight bit of change in the plan, now I need to use the sqlite cache in the machine to store the pending actions and events etc. So here is what happens inside calcachedcalendar.js all the sequence of events and functions that happens during the updation of items online or offline.

• When the calendar goes offline and online
• refresh() is called when it goes online after being offline before
• refresh() checks if the calendar is online, then it refreshes the network calendar first then calls the onLoad event where the cachedCalendar gets created again, otherwise, it calls the synchronize method which is responsible for making the offline an online sync changes.
• The synchronize() function, all the logic for the synchronization goes in basically the sync is between the cache stored on the machine and the network. first it takes in pending sync, does all the actions related to the pending sync and then it does ChangeLog calling the replayChangesOn method and lasting if everything fails, it goes and does the full sync, which essentially means throwing away the cache and then just downloading the changes from the network computer.
• When an addItem, adoptItem, deleteItem call is made it first gets routed through the cachedCalendar and then is called to the uncachedCalendar.
• mCachedCalendar pulls its information from the sqlite file. unCachedCalendar is on the network.

More details on the code changes soon!

cheers

msk

First Step : Getting event information

About Events in Calendar

Events in calendar inherit from the calIItemBase interface and in order to store them in a cached file i wud need to set up some sort of parsing from an event object to a serializable string which I can then write to a file.

As luck would have it there is an icalString attribute in calItemBase which allows you to get the event string in ical Format. I can store this guy and use it to construct a calItemBase object which I can then write to the cloud when online.

Here are the preliminary results.

http://mxr.mozilla.org/comm-central/source/calendar/base/public/calIItemBase.idl#64

The Great Plan

The Plan currently looks like this

So yeah this is the plan so far, Fallen and Ludovic have asked me to try and extend calCachedCalendar.js if possible because it would allow the providers to use the sync() functions for free without having to implement their own versions. 

So this is how I am going to approach the problem

• Add a property in IChangeLog interface, which also for offline sync and also stores a ChangeLog file, the implementation would be provided by different providers, and it is to be seen if I can move this inside calCachedCalendar.js but it won't make sense.
• Modify isCalendarWritable to see if the calendar can be synced offline which basically queries for IChangeLog and gets the offline sync property
• Add implementation for storing the sync actions in a flat file, for the CalDAV provider
• and implement a replayChangesOn -> generalized version to handle the two way sync

This week I will try to get the above set up done for at least the delete item method, coz its the easiest so far.

After that the second part would deal with the sync mechanisms and making it truly smart and worth having in the mozilla lightning trunk :-)

I guess I can say that I am quite confident of my understanding of the code in calUtils and calCachedCalendar

cheers

msk

Building Thunderbird + Calendar

Today I spent my entire time working with the building thunderbird and calendar together. I think I cleaned it about 3 times before I decided to backup my changes and update the directory from the start, that means, full clean-up. Everything deleted.

One of the important things I think is to keep updating the code on a daily basis so that you do not fall too behind the working set because it is for all these changes that the local code base falls behind and causes merging troubles.

I still not quite get the version systems CVS, HG, GIT, and others used by people, because it facilitates integration of code faster but it comes with its own set of headaches. Essentially, some body works on some part of the code at a single time, but I guess the versioning systems have their own advantages, especially git.

Long Night ahead, :-)

cheers
msk

Hacking around the code

First sucess is always a sweet one. No matter after how long it comes.

I spent this week, trying to hack around the calendar code-base.

Instead of making changes to the actual code and then building the product again and again, I am instead editing the application extension directly, since lightning is in the form of an extension to thunderbird, this makes it easy to edit the extension code and test out the functionality real time without the need for a build. This however, means, the application restarts everytime.

Anyways off to making at least deletion part of the syncing process.

cherios
msk

Mozilla Bug Assigned

Finally, I am down with the real stuff to be done. I have been assigned the bug in Bugzilla making me responsible for fixing up of the offline enhancement feature. It also puts a lot of community checks and balances since a number of people have an interest in the bug and would be making sure the fixes i propose are of a high standard.

Look up the details of the bug here: https://bugzilla.mozilla.org/show_bug.cgi?id=380060

Internet Protocols

Since starting off my GSoC assignment to get offline mode to work on the Lightning extension, there are a couple of things, that are needed to be studied in more careful details. One of them is the protocols that calendar providers use, in order to update calendars.

WebDAV is the base protocol over which a large number of other protocols such as CalDAV and CardDAV are based. HTTP itself does not collaboration among the authors of a document, it also does not say that it is impossible to do so.

As such a working group of the IETF drafted the WebDAV specs and they are being widely in the world right now in creation of newer methods of communication. It is being proposed that WebDAV become the standard of shared documents/resources on the web, thereby replacing FTP and other resource access protocols.

I was going through the specifications this week and found that WebDAV collaborators did not have any WebDAV enabled applications at that time, as such most of their work was based in using e-mails as a form of communicating changes in the document to a central author who then changed the document and mailed the document to the rest. Imagine, how painful it would be to mail everytime, and if the central author is not merging the changes (maybe he takes a break), the mechanism would certainly fail, causing trouble for many.

WebDAV Distributed Authoring Protocol defines a set of extensions to the base Hypertext Transfer Protocol for the capabilities like overwrite prevention, namespace management and properties.

I describe some details below:

• under overwrite prevention, WebDAV uses shared and exclusive locking mechanism, in order to prevent conflicts
• by properties, WebDAV protocol allows metadata to be stored and retrieved.
• namespace management, allows copying, moving and listing of shared resources, also known as collections.

Phew! that was simple. My next target is looking up at the specs of CalDAV and look up its implementation inside Lightning =)

cheers

mohit

The First Click

Yay, I have been selected as a participant of the Google Summer of Code Programme 2011. I will be working with Mozilla Calendar Project in order to extend the functionality of its lightning extension, so that it can support offline synchronization of the stored calendars of a person. This blog is going to be the officially maintained status report of the project. I am very excited to be part of the Mozilla community because it has always inspired me to take up software development. Jamie Zawinsky was involved in the foundation of Mozilla and spear headed the tirade of open source software. He is an inspiration for me and I hope to put my very best in this project.

Here's my proposal on the GSoC Website. http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/mohitkanwal/1

My Project Homepage on the GSoC Website: http://www.google-melange.com/gsoc/project/google/gsoc2011/mohitkanwal/25001

Let's Get Started for the ride :-)

msk