Ještě zajímavé info k technické části


Tech-heavy insight:
We managed to understand that many server calls were not being managed by RAM but were using an alternative data management method ("swap disk"), which is too slow for the flow of this amount of data. Once this data queued back too far, the service failed. Understanding why it was not using RAM was our key challenge and we worked with staff across multiple partners to troubleshoot this.
We spent over two days and nights applying numerous changes and improvement attempts: we both doubled the database servers and vertically scaled them by approximately 50% (“scale-up and scale out”). We re-balanced user profiles and inventories to new servers. Subsequent to the scale-up and scale-out, we also increased disk IOPS on all servers by approximately 40%. We also increased the headroom on the database, multiplied the number of shards (not the Anomalous kind) and continued to do all we were able to in order to force data into RAM.

Each of these steps helped us improve the resilience of the database when under extreme loads, but none of them were the "fix" we were looking for.

At this moment in time we are still waiting for a final Root Cause Analysis (RCA) from our partners, but ultimately what really helped resolve the overloading issue was configuring our database cache cleaning, which was being run every 60 seconds. At this frequency the database cache cleaning operation demanded too many resources which in turn led to the above mentioned RAM issues and a snowball effect that resulted in the connectivity issues seen.

We reconfigured the database cache cleanup operations to run more often with fewer resources, which in turn had the desired result of everything generally running at a very comfortable capacity.

All of this has enabled the servers to recover and sustain significantly more concurrent user loads.

Outriders

Multiplayer Stability and Connection:
Co-op and multiplayer sessions are a separate matter from solo play connectivity.
Launching a multiplayer session, requires multiple different partners and services. For example, matchmaking and queuing are hosted by one service partner, while the peer-to-peer relay is handled by another. Other components are hosted by other partners. This is a normal setup for multiplayer games but is important to mention because when we attempt to track multiplayer or co-op issues, knowing exactly what issue appears at what point is vital.

The upcoming patches should help alleviate crashes in multiplayer as well as improve stability and speed. Because of the complexity of the multiple systems involved, we will continue to improve and iterate upon these systems in order to deliver you a better multiplayer experience. In most cases, this can and has already been done via the backend without patching the game, but it is something we are keeping a close eye on.

btw. zjistil jsem, ze se mi furt vali v knihovne Remnant:From the Ashes. Urcite se tu najde nejakej harcovnik co hral oboji a dokazal by par slovy napsat nejaky srovnani:)

Kdyby nebylo těch lagů, tak sem si tu hru náramně užil. I když je příběh lehce děravej, tak má spád a vtánul mě do sebe. Ale to lagování ve 3 lidech mě málem donutilo s tou hrou přestat, protože se to prostě nedalo.

Když jsem hrál s Kidem duo, tak to občas zalagovalo, ale docela málo. Pak se k nám přidal kamarád ze Slovenska a tomu to lagovalo permanentně a mě začalo také. Takže to, že to nejede na jejich serverech je očividné ...

TRAGED: ale v zasade to odpovida tomu, co pisou llidi v commentech na Steamu

KID_MCHUTT: To mi chcers rict, ze MP je v tom udelanej tak, ze se jednoho stane server, jen routovanej pres jejich servery? To se mi nejak nezda.

WRATA: netuším, jak je to solo, kromě toho, že s pyro jsem měl problémy tam, kde jsem je předtím s techno neměl, ale dva a tři lidi jsou obtížností dost podobní, ten hlavní rozdíl je v tom, že ve třech to dvěma brutálně laguje a ve dvou to laguje jenom jednomu a ne moc.

KID_MCHUTT: Angry Joe si stezoval ze ve trech lidech je to silene nevybalancovane oproti tomu kdyz jdes solo nebo duo

tak jo, cca po 40 hodinách dohráno story.
- žádný nečekaný zvrat
- v posledních možná dvaceti videích totálně rozsynchronizovaný zvuk
- lagů jak nasráno, poslední boss díky tomu nešel zabít na 30/15, protože když člověka oneshotne neviditelný bordel a boss s každým dalším pokusem sype o jednu střílející chujovinu víc, nedá se to. Ani s full healing buildem, tankem a DD, které vydrží jak ženská.

je tam vidět, že to vydávali kurva narychlo, celkem dlouho před tím, než měli - ta poslední cca třetina vypadá nedodělaně a komplet neotestovaně. A to, že to při loadu najednou udělá brutálně hlasitý zvuk, který skoro urve hlavu, jen tak, protože může, je třešnička na dortu.
Jo a závěrečné titulky, stejně jako úvodní animace, nemají regulovaný zvuk, takže jedou na max. Asi aby na to lidi nekoukali. Viděli by, že tým víc šukal, než vyvíjel a testoval...

KID_MCHUTT: ono se to nastrada, kdyz ma jeden ten report kolem 300MB:p

GANDHI: 37Gb :-O

KID_MCHUTT: C:\Users\****\AppData\Local\Madness\Saved\Crashes

GANDHI: dík :) já tam totiž asi budu mít podobné obludy taky...

GANDHI: kam je to sype?

GANDHI: jo přes prime time to dneska drželo, ale lagovalo to jak prase... Jinak my už jedeme lvl 30/14 :)