Even if you don’t have any military goals you need a certain amount of Manpower – they are necessary in producing scout or colonizing ships. If you are out of Manpower you can’t build new ships and you can’t replenish your garrisons in systems or make invasions.
Ships with a full crew do better during space battles. The Manpower is required to operate your ships as its crew, they are the core of the defense and invasion forces. If you played the games from Europa Universalis series (or similar) you can quickly see the similarities. It determines how many soldiers you have on duty and in reserve.
Manpower is the numeric value represented by the icon with 3 people.