Web Browsers have been built to render web pages with reference to W3C recommendations. In old days, web pages were typically built for Netscape navigator and Internet Explorer. But when web standards made at W3C then web pages of existing site start breaking because browsers could not start using that standards. So browsers comes up with 2 modes one for new standard compliant sites and other for old legacy sites.
Now modern browsers comes with 2 modes quirks mode and standard or strict mode.
Quirk mode is not any standard , when any web page does not have a Doctype then browser render a web page in Quirk mode or quirk mode is turned on when there is no correct Doctype declaration in any web page and auto turned off after adding valid Doctype.
Read more about HTML Doctypes
For internet explorer, if an html page contains valid <!Doctype> then IE uses one of the standard document mode and if there is not valid Doctype then it uses Quirk mode. In internet explore, there is option to change modes.
So, how browser know which mode to use – Browsers uses the DOCTYPE in web page to determine whether to handle in quirk mode or standard mode. Always make sure that web page should have valid Doctype to use standard mode.
Above example using HTML5 Doctype, the current standard and recommended by HTML5. Earlier versions of HTML recommend other variants of standard but now all existing browsers are using full standard mode for the DOCTYPE.