Things don't start getting weird until you start moving stuff around using all your fancy CSS tools. A plain, naked, HTML site is perfectly responsive, and honestly it has a kind of nostalgic appeal.
In the real world, however, websites just can't be this plain.
Users will think it's broken or cheap. Usability is a concern for anything that goes beyond simple informational sites. Even Wikipedia, as plain as it seems, uses CSS to create headers, footers and sidebars