java.lang.OutOfMemoryError: Java heap space

What it means

The server asked Java for more memory and there was none left inside the heap it was given at startup. This is not about how much RAM your machine has - it is about the ceiling set by the -Xmx flag the server was launched with. Once the server reaches that ceiling the JVM throws this error, and the server usually dies within seconds.

It shows up in one of two situations: the allocation was simply too small for the pack and player count, or a mod is leaking memory and would eventually fill any allocation you gave it.

How to fix it

• Raise the allocation. Rough guide: 2-3 GB for vanilla with a few friends, 4-6 GB for a light mod list, 8 GB or more for a large modpack. In Coalstack that is the RAM setting on the Dashboard's Overview page; by hand it is the -Xmx6G value in your start script.
• Do not allocate everything. Leave at least 2 GB for the operating system. Giving an 8 GB machine an 8 GB heap makes crashes more likely, not less.
• Check whether it climbs back up. If the server runs out again a few hours after every restart no matter how much you give it, that is a leak in a mod rather than an allocation problem. A healthy server's memory use sawtooths as garbage collection runs; a leaking one only ever goes up.
• Suspect the heavy mods first. World-generation mods, dynamic maps, and chunk pregenerators are the usual causes. Remove recently added ones one at a time.

If you are running 32-bit Java you cannot allocate more than roughly 1.5 GB no matter what you set - install a 64-bit JDK, which the Java version guide walks through.