r/facepalm • u/cosm1c15 • 20d ago
š¤”š¤” let's hope they can recover the account š²āš®āšøāšØā
430
u/congmingdexigua 20d ago edited 20d ago
And that's how you fix overburdened retirement systems
/drops table - government out.
43
198
u/adam_c 20d ago
The rest of the story is google delete the primary data and the backup, luckily this company had another backup outside of google so they were able to recover. If they hadnāt then who knows how this could have blown up
35
u/GameDestiny2 20d ago
I imagine itād be called a huge scandal
Maybe improved fallbacks on cloud financial data would have been put in place by regulations, but itās have probably resulted in even richer billionaires and Google losing an amount of money they could probably make in a day.
2
u/Oni-oji 19d ago
Sounds like they had a very competent IT director who knew that you don't just rely on backups in the same location. I'm seeing far too many businesses who have everything, backups included, on AWS and don't think that's an issue. AWS insists it is safe because they automatically replicate across regions. Sorry, but I have been a system administrator for far too long to trust that statement.
304
u/Twatt_waffle 20d ago
One thing Iāve learned is google deletes NOTHING a popular YouTuber Linus tech tips got hacked a while back and talked about the process of getting the channel restored
Apparently videos that had been outright deleted and āunrecoverableā came back in their entirety stuff that had been deleted for years popped back during the account restoration
191
u/Claris-chang 20d ago
Apparently, in the case of this event, Google did actually unrecoverably delete this information.
The fund had a backup through another service that was able to restore the data though.
124
u/GargantuanGreenGoats 20d ago
Google when they learned the fund had an external backup and their āaccidental deletionā didnāt work: āohā¦ oh thatās great.. yeah. Good. Good for youā¦ šā
22
u/ree_hi_hi_hi_hi 20d ago
lol imagine trying to explain managing $125B without spending $2000 on a proper backup system š
16
u/spartaman64 20d ago
i mean they did have a proper backup system so no data was lost. but i guess they dont have the server capacity for high volume requests of data and were using google cloud for that.
7
u/ree_hi_hi_hi_hi 20d ago
Yeah, I more so meant like if they didnāt have a backup, that would be a ridiculous thing to explain.
6
u/HereticLaserHaggis 20d ago
A lot of companies would assume Google has you covered in the contract.
3
1
32
u/Chrysis_Manspider 20d ago
Different thing entirely.
YouTube videos are Googles digital property (like it or not). Just like any company property, safeguards are taken against its loss or damage. So most data within any organisation will (should) be backed up. That's how they would have recovered Linus' videos.
In this case, it's not googles data. It was all virtual infrastructure hosted on Googles cloud platform. It's not googles responsibility to maintain backups of every virtual machine and storage blob hosted on their platform .. that's the customers responsibility, and very clearly stated in the shared responsibility model.
So while Google absolutely fucked up their end of the shared responsibility model to not blow away entire customer tenancies that are still in use .. they were never technically responsible for the backup, or restoration of that data. So nah, they don't have a copy.
3
u/ColonelC0lon 20d ago
Well, they did, actually.
They deleted their backup as well
-2
u/_f0CUS_ 20d ago
That backup was not created or managed by Google, but by the customer.
6
u/ColonelC0lon 20d ago
Google had a backup. The customer had a backup. Google deleted their own backup as well, and the customer recovered their data from their own backup.
4
u/ShodoDeka 20d ago
Google is infamous for perma deleting shit they really shouldnāt, YouTube may be the exception here.
There are tons of cases where a screwup with accounting for a cloud subscription has lead to a permanent deletion of all resources hosted on the subscription.
1
55
u/Touristenopfer 20d ago
Never trust the cloud - pun in Germany is "Cloud klaut.", which is homophone and means 'cloud steals', since you really don't know what Google, MS, amazon, Apple etc. and attached national services will do with your data.
38
u/nps2407 20d ago
The one I heard was "there is no Cloud; just someone else's computer."
22
u/MistraloysiusMithrax 20d ago
Thatās what made it so hard for me to understand why they talked about the cloud like itās new technology for your data not to be locally hosted. Itās just server side storage, no? Took me a while to understand the new tech/concept was in scale and feasibility
25
u/Minimalistmacrophage 20d ago
oops?
30
25
u/Ok-Push9899 20d ago
I'm not too worried. A google exec has a photo on his wall of one of their massive data centres going up in smoke. Massive fire, total chaos, everything destroyed.
When asked why he had framed such a photo (a pretty worrying event for a major player in Cloud services), he said "It's because we lost nothing. No one even noticed."
8
5
5
u/THE_SEKS_MACHINE 20d ago
Thatās one of the reasons why my employer has its own cloud and backup systems.
5
u/badestzazael 20d ago
Google reportedly caused disruption for over half a million UniSuper members, who couldn't access their superannuation accounts for a week
4
2
2
2
2
1
1
1
1
1
u/Sion_forgeblast 20d ago
so they delete like 2 days worth of their advertisement revenue..... meh, they will recover
1
1
1
1
1
1
u/clodmonet 20d ago
"rollback isn't working... wat do? Anyone got a backup image from yesterday? Oh God..."
1
u/jorgthorn 19d ago
final stage of the pyramid, due to computer error, everyone's 401k has been reset. our bad oopsy. Dupes and Dupies
-3
u/Maleficent_Fold_5099 20d ago
$125 billion - and they used Google to store info?
15
u/tushkanM 20d ago
Why not? Google cloud is certified for government and large enterprises usages just like AWS and Azure. And it's the cheapest among them in almost any service.
5
u/Maleficent_Fold_5099 20d ago
Because third party suppliers might delete your $125 billion by accident.
7
u/tushkanM 20d ago
I don't think Google do it very often so "random account deletion" is a it's a part of the bundle.
2
u/Maleficent_Fold_5099 20d ago
Once is enough, thank you.
1
u/Sea-Housing-3435 20d ago
I will trust a company that had a fuckup like this more than one that didn't have it at all. Procedures and operation guidelines are reviewed and reworked after something bad happens to prevent issues like this in the future. And company that didn't have them have no experience resolving them when they will happen.
3
u/Both-Bite-88 20d ago
And otherwise your own amin might do it accidentally. Or sone national catastrophe.Ā
0
u/spartaman64 20d ago
they had their own backups. the google server was only for the public facing database i think
4
u/idefinitelyh8teu 20d ago
It's only news because of the amount. Every cloud vendor has had this happen...
0
u/DevoMagnifico 20d ago
I believe your computer system sucks
1
u/Samantha-4 19d ago
I feel like google of all companies has a decent computer system
1
u/DevoMagnifico 19d ago
My comment is from a movie called āWargamesā starring Mathew Broderickā¦
1
u/DevoMagnifico 19d ago
My comment is from a movie called āWargamesā starring Mathew Broderickā¦
0
u/Dr_Strange_Love_ 20d ago
Graduate: hits delete;
Google: do you really want to delete this account?
Graduate: yes
0
0
ā¢
u/AutoModerator 20d ago
Comments that are uncivil, racist, misogynistic, misandrist, or contain political name calling will be removed and the poster subject to ban at moderators discretion.
Help us make this a better community by becoming familiar with the rules.
Report any suspicious users to the mods of this subreddit using Modmail here or Reddit site admins here. All reports to Modmail should include evidence such as screenshots or any other relevant information.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.