First at all thank you Packet.net and AMD for the Challenge and the free time with the AMD EPYC or c2.medium.x86 how packet.net calls it.
Befor i tell you the story i tell you what?!? Always pushing the boundaries! Benchmark
The main Idea was to repeat the ARM64 Project. But due to the limited amount of RAM (only 64GB instead of 128GB) I was not able to do that exact same messurement.
The Server startup was a very critical Moment every time. I started 50 Servers as fast as Docker can spawn the containers. Either it was successful but slow or the Kernel killed some/most processes due to the fact that the Memory and/or Swap was near very critical limits. Every time i was greeted with a nice 4.7 GB core dump, for every crashed Docker container 😀
[18401.493117] CPU: 33 PID: 51489 Comm: java Tainted: G L 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
[18401.493123] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.3.6 04/20/2018
[18401.493125] task: ffff8991541fb000 task.stack: ffff9d3661098000
[18401.493138] RIP: 0010:[<ffffffffab72993d>] [<ffffffffab72993d>] dump_stack+0x35/0x78
[18401.493142] RSP: 0018:ffff9d366109bba0 EFLAGS: 00000286
[18401.493145] RAX: 000000000000001a RBX: 0000000000000286 RCX: 00000000ffffffff
[18401.493152] RDX: 0000000000000021 RSI: 0000000000000296 RDI: 0000000000000286
[18401.493160] RBP: ffff9d366109bc30 R08: 0000000000000000 R09: 0000000000000031
[18401.493164] R10: 0000000000000000 R11: 00000000000b3498 R12: 0000000000000001
[18401.493168] R13: ffff89973ff7bcc0 R14: ffff89973ff7bd20 R15: ffff9d366109bd68
[18401.493175] FS: 00007ff3e4cc1ae8(0000) GS:ffff898f2f800000(0000) knlGS:0000000000000000
[18401.493184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[18401.493189] CR2: 0000000000818d58 CR3: 0000000ed332a000 CR4: 00000000003406f0
[18401.493189] Stack:
[18401.493202] ffffffffabdfeae0 ffff9d366109bc30 ffffffffab58569a 024280ca93c1e475
[18401.493211] ffffffffabdfeae0 ffff9d366109bbd0 0000000000000010 ffff9d366109bc40
[18401.493223] ffff9d366109bbf0 c1b0f64b93c1e475 00000000000000f0 0000000000000013
[18401.493225] Call Trace:
[18401.493239] [<ffffffffab58569a>] ? warn_alloc+0x13a/0x160
[18401.493242] [<ffffffffab5860c5>] ? __alloc_pages_slowpath+0x995/0xbf0
[18401.493257] [<ffffffffab57e3e6>] ? wait_on_page_bit_killable+0x96/0xa0
[18401.493267] [<ffffffffab58651e>] ? __alloc_pages_nodemask+0x1fe/0x260
[18401.493273] [<ffffffffab5d86ce>] ? alloc_pages_vma+0xae/0x260
[18401.493290] [<ffffffffab5b4156>] ? handle_mm_fault+0x10b6/0x12d0
[18401.493303] [<ffffffffab424761>] ? __switch_to+0x2c1/0x6d0
[18401.493313] [<ffffffffab45ee5c>] ? __do_page_fault+0x25c/0x500
[18401.493323] [<ffffffffaba08b58>] ? page_fault+0x28/0x30
[18401.493477] Code: 00 00 48 89 c3 fa 66 0f 1f 44 00 00 65 8b 15 c3 38 8e 54 89 c8 f0 0f b1 15 41 7a 98 00 83 f8 ff 74 12 39 c2 74 12 48 89 df 57 9d <0f> 1f 44 00 00 f3 90 eb c7 31 ed eb 05 bd 01 00 00 00 48 c7 c7
[18401.516890] NMI watchdog: BUG: soft lockup - CPU#37 stuck for 22s! [java:51270]
[18401.516973] Modules linked in: xt_nat xt_tcpudp veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay nls_ascii nls_cp437 vfat kvm_amd mgag200 fat efi_pstore ttm dcdbas kvm drm_kms_helper irqbypass evdev drm pcspkr ccp crct10dif_pclmul efivars crc32_pclmul sg i2c_algo_bit rng_core ghash_clmulni_intel ipmi_si shpchp sp5100_tco ipmi_msghandler tpm_crb acpi_power_meter button dm_service_time dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua tcp_westwood ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bonding efivarfs ip_tables x_tables autofs4 ext4 crc16
[18401.517055] jbd2 fscrypto ecb mbcache raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod sd_mod xhci_pci xhci_hcd crc32c_intel ahci libahci aesni_intel aes_x86_64 glue_helper mpt3sas mlx5_core lrw raid_class gf128mul ablk_helper devlink scsi_transport_sas libata cryptd usbcore ptp scsi_mod i2c_piix4 usb_common pps_core
Host dmesg (There were thousands of this messages)
The problem was thus not only the overwhelmed memory with my idea instead it also was the CPU which limited the project.
But i would not be called the green light htop master (Thanks Jacob 😉) when i would not be able to find a solution for that.
I repeated it with 45, 40, .. and with 20 and 4 GB of ram i had a good success rate. Because for unknown reasons the Java garbage collector was not so happy. I was forced to pimp up the memory otherwise the map generation process stopped after some time and waited till the memory was free. Not that good for a speed test.
What i came up with what that the server startup was very CPU hungry and the map generation not. It is throttled by default to 20 chunks per second so i started the servers step by step and very slowly. This is what i got.
[00:21:35 INFO] [WorldBorder] Set default border shape to rectangular/square.
[00:21:35 INFO] [WorldBorder] [CONFIG] Configuration saved.
[00:21:35 INFO] World generation task is ready for world "world", attempting to process up to 20 chunks per second (default 20). The map will be padded out 208 blocks beyond the border (default 208). Parts of the world which are already fully generated will be skipped.
[00:21:35 INFO] This process can take a very long time depending on the world's border size. Also, depending on the chunk processing rate, players will likely experience severe lag for the duration.
[00:21:35 INFO] You should now use wb fill confirm to start the process.
[00:21:35 INFO] You can cancel at any time with wb fill cancel, or pause/unpause with wb fill pause.
[00:21:35 INFO] [WorldBorder] world: world padding: 208 repeats: 1 ticks: 1
[00:21:35 INFO] WorldBorder map generation task for world "world" started.
[00:21:40 INFO] [WorldBorder] [Fill] 632 more chunks processed (632 total, ~0.5%) (free mem: 603 MB)
[00:21:45 INFO] [WorldBorder] [Fill] 96 more chunks processed (728 total, ~0.6%) (free mem: 386 MB)
[00:21:50 INFO] [WorldBorder] [Fill] 99 more chunks processed (827 total, ~0.7%) (free mem: 581 MB)
[00:21:55 INFO] [WorldBorder] [Fill] 98 more chunks processed (925 total, ~0.8%) (free mem: 349 MB)
#Snip
[01:56:21 INFO] [WorldBorder] [Fill] 100 more chunks processed (114164 total, ~97.9%) (free mem: 875 MB)
[01:56:21 INFO] [WorldBorder] [Fill] Saving the world to disk, just to be on the safe side. (free mem: 875 MB)
[01:56:25 INFO] [WorldBorder] [Fill] 1777 more chunks processed (115941 total, ~99.4%) (free mem: 640 MB)
[01:56:25 INFO] [WorldBorder] [Fill] task successfully completed for world "world"! (free mem: 629 MB)
Generation of 2500x2500 Map
Generation of 2500x2500 Map
The green light
After that i concluded that…
Next idea was to check how long it would take to generate one single map.
I gave the server 60 Gigs of RAM and instructed WorldBoard to generate a maximum of 1000 Chunks per second. If the System could deliver that amount of power we would finish the map generation after 104 Seconds! In numbers this is ~2500x2500 = 170 Chunks (WorldBoards adds 208 Blocks so that the map does not end in nowhere) Which is impressive if we remember that a chunks consists of 16x16 Blocks with a hight of 256.
The generation needed 20 Minutes. So 27082708 * 256 => *1.877.315.584 Blocks (with air) The snow level is around 90 block height. Even if we use this as an average number we still generated 659.993.760 Blocks.
And we only needed 20 Gigs of RAM at the end 😀
[20:17:39 INFO] Done (19.973s)! For help, type "help" or "?"
[20:17:39 INFO] [WorldBorder] Border set. World "world" has border radius 2500 at X: -248.0 Z: 64.0
[20:17:39 INFO] [WorldBorder] [CONFIG] Configuration saved.
[20:17:39 INFO] Border has been set. World "world" has border radius 2500 at X: -248.0 Z: 64.0
[20:17:39 INFO] [WorldBorder] Set default border shape to rectangular/square.
[20:17:39 INFO] [WorldBorder] [CONFIG] Configuration saved.
[20:17:39 INFO] World generation task is ready for world "world", attempting to process up to 1000 chunks per second (default 20). The map will be padded out 208 blocks beyond the border (default 208). Parts of the world which are already fully generated will be skipped.
[20:17:39 INFO] This process can take a very long time depending on the world's border size. Also, depending on the chunk processing rate, players will likely experience severe lag for the duration.
[20:17:39 INFO] You should now use wb fill confirm to start the process.
[20:17:39 INFO] You can cancel at any time with wb fill cancel, or pause/unpause with wb fill pause.
[20:17:39 INFO] [WorldBorder] world: world padding: 208 repeats: 50 ticks: 1
[20:17:39 INFO] WorldBorder map generation task for world "world" started.
[20:17:44 INFO] [WorldBorder] [Fill] 832 more chunks processed (832 total, ~0.7%) (free mem: 52653 MB)
[20:17:49 INFO] [WorldBorder] [Fill] 416 more chunks processed (1248 total, ~1.1%) (free mem: 53530 MB)
[20:17:54 INFO] [WorldBorder] [Fill] 441 more chunks processed (1689 total, ~1.4%) (free mem: 52151 MB)
[SNIP]
[20:37:15 INFO] [WorldBorder] [Fill] Saving the world to disk, just to be on the safe side. (free mem: 35754 MB)
[20:37:20 INFO] [WorldBorder] [Fill] 523 more chunks processed (113623 total, ~97.4%) (free mem: 34653 MB)
[20:37:25 INFO] [WorldBorder] [Fill] 471 more chunks processed (114094 total, ~97.8%) (free mem: 33211 MB)
[20:37:29 INFO] [WorldBorder] [Fill] 3213 more chunks processed (117307 total, ~100.0%) (free mem: 31769 MB)
[20:37:29 INFO] [WorldBorder] [Fill] task successfully completed for world "world"! (free mem: 31769 MB)
Generation of 2500x2500 Map #
Next project was a the World of Warcraft Azeroth Map in Minecraft which my Friend Georg suggested me.
It was rendered in around 40 Minutes. Which resulted in 431603 Folders, 1690784 .png files which occupied 11 GB on the raid 0. Sadly the map was rendered incorrectly because it was to high.
We can remove ~1 Minute because 7z is indexing the file with only one core 😔
root@Wacan:/data# time 7z a -mmt=on -mmt=48 -md=96m -mfb=256 map.7z map/
7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,48 CPUs AMD EPYC 7401P 24-Core Processor (800F12),ASM,AES-NI)
Scanning the drive:
431603 folders, 1690784 files, 11712674430 bytes (11 GiB)
Creating archive: map.7z
Items to compress: 2122387
Files read from disk: 1690784
Archive size: 9615765537 bytes (9171 MiB)
Everything is Ok
real 10m33.738s
user 107m27.108s
sys 1m19.288s
I hope you learned something from this.