Memory Bottlenecks

On a Linux system, many programs run at the same time. These programs support multiple users, and some processes are more used than others. Some of these programs use a portion of memory while the rest are “sleeping.” When an application accesses cache, the performance increases because an in-memory access retrieves data, thereby eliminating the need to access slower disks.

The OS uses an algorithm to control which programs will use physical memory and which are paged out. This is transparent to user programs. Page space is a file created by the OS on a disk partition to store user programs that are not currently in use. Typically, page sizes are 4 KB or 8 KB. In Linux, the page size is defined by using the variable EXEC_PAGESIZE in the include/asm-/param.h kernel header file. The process used to page a process out to disk is called pageout.

Finding Memory Bottlenecks

Start your analysis by listing the applications that are running on the server. Determine how much physical memory and swap each application needs to run.

Memory available:-

This indicates how much physical memory is available for use. If, after you start your application, this value has decreased significantly, you might have a memory leak. Check the application that is causing it and make the necessary adjustments. Use free -lt for additional information.

Page faults:-

There are two types of page faults: soft page faults, when the page is found in memory, and hard page faults, when the page is not found in memory and must be fetched from disk. Accessing the disk will slow your application considerably. The sar -B command can provide useful information for analyzing page faults, specifically columns pgpgin/s and pgpgout/s.

File system cache:-

This is the common memory space used by the file system cache. Use the free -lt command for additional information.

Private memory for process:-

This represents the memory used by each process running on the server. You can use the pmap command to see how much memory is allocated to a specific process.

Why swap is getting used though there is enough memory?

Linux handles swap space efficiently. While swap is nothing more than a guarantee in case of over-allocation of main memory in other operating systems, Linux kernel utilizes swap space far more efficiently. Virtual memory is composed of both physical memory and the disk subsystem or the swap partition. If the virtual memory manager in linux realizes that a memory page has been allocated but not used for a significant amount of time, it moves this memory page to swap space.

Often you will see daemons such as "getty" that will be launched when the system starts up but will hardly ever be used. It appears that it would be more efficient to free the expensive main memory of such a page and move the memory page to swap. This is exactly how Linux kernel handles swap, so there is no need to be alarmed if you find the swap partition filled to 50%. The fact that swap space is being used does not mean a memory bottleneck but rather proves how efficiently Linux handles system resources. Also, a swapped out page stays in swap space until there is a need for it, that is when it gets moved in (swap-in).

Why is there a high amount of cache memory being consumed? What is this “cached/cache” in free command output?

Simply it shows the amount of RAM space occupied by "Page Cache". Cache pages are nothing but the data/files which gets copied over to RAM when kernel performs read/write operation on a disk. The reason for keeping these page cache is for I/O performance. So, kernel maintains these files on RAM and frees them up whenever not required or whenever a new memory space is requested by a new process/command provided no free space available in RAM. For a simple read operations it would be marked as "clean" pages, however, writes would mark them "dirty" and becomes clean after updating on disk. These "cached" pages would remain in memory even not used and would not move out even when not required. So, as a system usage goes these cached pages size would also grow. This is usually not a concern to be worried, however, sometimes yes when "cache" is consuming most of the memory and system starts to swap pages.

Performance tuning options:-

- Tune the swap space using bigpages, hugetlb, shared memory.
- Increase or decrease the size of pages.
- Improve the handling of active and inactive memory.
- Adjust the page out rate.
- Limit the resources used for each user on the server.
- Stop the services that are not needed.
- Add memory.

Tuning the vm subsystem:-

Tuning the memory subsystem is a challenging task that requires constant monitoring to ensure that changes do not negatively affect other subsystems in the server. If you do choose to modify the virtual memory parameters (in /proc/sys/vm), we recommend that you change only one parameter at a time and monitor how the server performs.

Remember that most applications under Linux do not write directly to the disk; they write to the file system cache maintained by the virtual memory manager that will eventually flush out the data. When using an IBM ServeRAID controller or an IBM TotalStorage disk subsystem, you should try to the decrease the number of flushes, effectively increasing the I/O stream caused by each flush. The high-performance disk controller can handle the larger I/O stream more efficiently than multiple small ones.

Setting kernel swap and pdflush behavior:-

The parameter stored in /proc/sys/vm/swappiness can be used to define how aggressively memory pages are swapped to disk. An introduction to the Linux virtual memory manager and the general use of swap space in Linux is discussed in “Page frame reclaiming” on page 14. It states that Linux moves memory pages that have not been accessed for some time to the swap space even if there is enough free memory available. By changing the percentage in /proc/sys/vm/swappiness you can control that behavior, depending on the system configuration. If swapping is not desired, /proc/sys/vm/swappiness should have low values. Systems with memory constraints that run batch jobs (processes that sleep for a long time) might benefit from an aggressive swapping behavior. To change swapping behavior, use either echo or sysctl as shown below.

#Changing swappiness behavior
[root@server ~]# sysctl -w vm.swappiness=100

Especially for fast disk subsystems, it might also be desirable to cause large flushes of dirty memory pages. The value stored in /proc/sys/vm/dirty_background_ratio defines at what percentage of main memory the pdflush daemon should write data out to the disk. If larger flushes are desired then increasing the default value of 10% to a larger value will cause less frequent flushes. As in the example above, the value can be changed as shown below.

Increasing the wake up time of pdflush
[root@server ~]# sysctl -w vm.dirty_background_ratio=25

Another related setting in the virtual memory subsystem is the ratio at which dirty pages created by application disk writes will be flushed out to disk. The writes to the file system will not be written instantly but rather written in the page cache and flushed out to the disk subsystem at a later stage. Using the parameter stored in /proc/sys/vm/dirty_ratio the system administrator can define at what level the actual disk writes will take place. The value stored in dirty_ratio is a percentage of main memory. A value of 10 would mean that data will be written into system memory until the file system cache has a size of 10% of the server’s RAM. As in the previous two examples, the ratio at which dirty pages are written to disk can be altered as follows to a setting of 20% of the system memory:

Altering the dirty ratio
[root@server ~]# sysctl -w vm.dirty_ratio=20

Swap partition:-

The swap device is used when physical RAM is fully in use and the system needs additional memory. Linux also uses swap space to page memory areas to disk that have not been accessed for a significant amount of time. When no free memory is available on the system, it begins paging the least used data from memory to the swap areas on the disks. The initial swap partition is created during the Linux installation process with current guidelines stating that the size of the swap partition should be two times physical RAM. Linux kernels 2.4 and beyond support swap sizes up to 24 GB per partition with an 8 TB theoretical maximum for 32-bit systems. Swap partitions should reside on separate disks.

If more memory is added to the server after the initial installation, additional swap space must be configured. There are two ways to configure additional swap space after the initial install:

1. A free partition on the disk can be created as a swap partition. This can be difficult if the disk subsystem has no free space available. In that case, a swap file can be created.

2. If there is a choice, the preferred option is to create additional swap partitions. There is a performance benefit because I/O to the swap partitions bypasses the file system and all of the overhead involved in writing to a file.

How to test all above tips at home for Memory issue.

"stress" tool

It is a workload generator tool designed to subject your system to a configurable measure of CPU, memory, I/O and disk stress.

DO NOT TRY THIS TOOL IN PRODUCTION.