Yes, it matters.
What differs for any particular lot is the degree to which it matters.
You might end up with a lot in which some elements are missing (for example, carpets or tile).
You might end up with a lot in which some elements (for example, windows) show up as a Maxis default or otherwise incorrectly.
You might end up with a lot in which some elements are flashing blue (or else show up as strings of text and numerals due to a missing mesh).
You might end up with a lot which either will not load or which will crash your game.
You might end up with a lot which contains game-wide default replacements (and thus will change other lots you already have).
It all depends on what EP- or SP-dependent elements are included with the lot.
Using Clean Installer to install lots is highly recommended, as it gives you the option of selecting what you actually want to install (the barebones lot/house and/or any other particular elements which may be included with it) as well as the option to not install any hacks which might (advertantly or inadvertantly) be included with the lot (and any of which might negatively impact your game). It is not uncommon for lot creators to create a house/lot in their game set-up which includes hacks (either minimal or global hacks) or hacked objects and either be unaware of that or fail to provide that information in their descriptions.
Note that Clean Installer will not give you the 'missing EPs/SPs' warning, but on the other hand will allow you to check many items individually and have the install process ignore additional associated content which may be EP- or SP-dependent.
Info/download of Clean Installer here:
http://sims2pack.modthesims2.com/index.php?page=4