Occasionally, after coming back to a Kinopio space I had been working in, I observe a card has disappeared. I usually detect this by seeing a connection going to nowhere. I don’t have reproduction steps.
In one case, I first noticed the missing card on a freshly loaded browser. When I went to the browser where I originally created the card and loaded the space, I briefly saw the card. But then it disappeared. That seems to indicate that it was in my localStorage, but then synced with the server and the card was obliterated.
I had an occurrence which I documented in Slack: Slack
Some miscellaneous facts:
I’ve checked localStorage.queue in these situations, and it has always been empty.
The card is not in my Removed cards list.
This has happened on a variety of spaces (Private, Closed), in spaces that have been shared and not shared.
I observe a card has disappeared. I usually detect this by seeing a connection going to nowhere.
Do you remember the card? Is it possible that the card isn’t missing, but that the card was removed and the connection lines getting subsequently removed is actually the change that didn’t sync?
(not meaning to gaslight, just troubleshooting )
I added additional user visible connection error logging, and connecting states a couple days ago to help address this. If you see this issue again it’d be a good clue if I knew if you let me know what error notifications you see
I saw this again in one of my spaces. I don’t have any logs or error messages to report, which is kind of frustrating. Usually what happens is, after coming back to a space I’ve been working in, I notice a connection to nothing. I check the console logs and don’t see any errors. I check the localStorage.queue – nothing. This also seems to happen in spaces where I’m the only user (no collaborators.)
Do you have any other ideas of logging or ways to troubleshoot this? I am confident I have created the card, but somehow it doesn’t make it to the server and it gets blown away. Is there a way on the server you can detect this situation, given many times there’s a connection to that missing card? Wouldn’t that mean the connection would have an id to the missing card?
Not trying to pile on a problem you’re already aware of, but it’s concerning because of the data loss
You may have explained this before already, and thanks for bearing with me since I don’t really know the underlying API, but some questions:
How would a card get created and a connection to that card get created, but then the card is not on the server, but the connection is?
If the card is only getting created locally and for some reason that message never makes it to the server, why is the server clobbering the card? There is no explicit user-initiated removal of the card, so can the clobbering not be allowed?
If the problem is the create card message not making it to the server, how/why is that message disappearing without a trace? Can you add visibility/logging around that?
How would a card get created and a connection to that card get created, but then the card is not on the server, but the connection is?
what I see that was happening is that the card creation request is hitting the server and then failing to create the card in the db, sometimes because of a ‘foreign key constraint’ error, but I think that specific error is a red-herring. The larger theme is that some db writes are being rejected, for cards only.
If the card is only getting created locally and for some reason that message never makes it to the server, why is the server clobbering the card? There is no explicit user-initiated removal of the card, so can the clobbering not be allowed?
Assuming the system was working 100%, the replacing local with server shouldn’t result in missing data. The issue with not clobbering a local-only card is trading the upfront pain of losing a card vs the much-harder-to-diagnose longterm pains of your space being out of sync with the-source-of-truth (server). In that scenario, if a sync error happens and the card never gets added to the db but is still on your local, it’ll cause a series of cascading errors whenever you attempt to do any operation with that card, like connecting things to it, or collaborating.
The latter issues may all be bandaid fixable, but imo it’s better to fix the source of the issue: the db faliing to create cards in the first place.
If the problem is the create card message not making it to the server, how/why is that message disappearing without a trace? Can you add visibility/logging around that?
There is currently logging around it, I haven’t been found the time to investigate it more thoroughly yet. In the meantime, keep letting me know when you’re seeing this issue because that’ll help me grab logging timestamps.
After tags is complete, my plan is to spend a week or two on fixing this issue and some other performance improvements.
I updated my logging system over the weekend to give me cleaner outputs and to better isolate error messages. This is an example db error causing a card to not be created
{
"time": "04:47pm",
"message": "[GREEN] [10-5] \thttps://kinopio.club/-CARDURL?hidden=true',false,'2020-09-20 16:47:54.294 +00:00','ID','ID','2020-09-20 16:47:54.295 +00:00','2020-09-20 16:47:54.295 +00:00') RETURNING \"id\" INTO primary_key; created := true; EXCEPTION WHEN unique_violation THEN UPDATE \"cards\" SET \"id\"='CARDID',\"z\"=41,\"spaceId\"='SPACEID',\"name\"='🌷SPACENAME"
}
]
from the logs, as you’d expect, it’s a pretty rare occasion, less than once a day. but def still needs to be fixed
Occurred within this hour: Sep 22, 2020 3:18 EDT - 4:20 EDT
I’m curious, does this match up with a log on your end?
Something weird is that the name is blank on that card. I think it was Fruitvale Station, but that value was never in the hourly backups I’ve been doing.
so the card id with XYrc5j9wkbt2T3I-XElcg is not in the db or a card with the name Fruitvale Station
Looking around the 3pm-4pm EDT mark (7-8pm UTC) on the 22nd I don’t see anything out of the order.
The server changes for tagging that are upcoming include some extra error logs (including logging item IDs) that’ll help with debugging this issue in the future
I know that the card (XYrc5j9wkbt2T3I-XElcg) was created sometime between 12:33 pm to 2:20pm on Sep 22 (EDT). I started doing my hourly backups, so it certainly was on the server at some point because at 2:20pm, I did a GET via the API. It was still there at 3:20pm. But then it was gone at the 4:20pm GET.
And the connection (IGhf0rJYkK-8d2S-6zGUf) has its endCardId set to the missing card.
hihi, i just pushed a little db fix for card creating which i’m hoping addresses this problem. I can’t repro it easily so I’ll leave this open for now – Let me know if you lose cards again. thx
oh nm , that didn’t fix it but it helped me find the real problem (a different race-condition, of course). Hang tight, I should have a fix hopefully today or tomorrow
OK added some race-condition mitigations that should help:
Now, when you load a space, if you have a card in your cache that isn’t present on the server space, I’ll now double check the server to see if that card exists. If the card exists, I’ll add it back to the space shown. The card may not of originally existed on the returned space because it’s was still being written to the db.
Additionally, cards that are removed due to sync won’t be permanently removed. They’ll be recoverable from ‘restore’ if they were given a name.
I’m pretty confident about this fix, let me know if you still see missing cards
)
(-‿-)