Understanding the World Pixel by Pixel