My boyfriend's brother lives with us, doesn't pay rent, parties all night & sleeps in all day. ?

My boyfriend asked me nine months ago if it was okay if his brother could stay with us till he got on his feet. I said as long as it's just a short time, couple months, until he found a job. He has partied the whole time, comes in late, sleeps past noon, and now he has a job, but still doesn't pay rent & barely helps around the house. I'm supposed to be quiet in the mornings so I don't wake him. My boyfriend says I'm too loud and that I should be more respectful but I don't feel I'm getting any respect because my boyfriend said he built this house for us and our three kids. His son has a bedroom, his brother has a bedroom & my two children have to sleep together on an air mattress. I used a credit card to buy close to $10,000 in furniture which should be the equivalent of $1100 in rent monthly. I just started a business & I still help around the house. I feel taken advantage of & disrespected. My boyfriend says he doesn't want to have to chose between his love & blood but it's not fair to my children or to me. Am I wrong in feeling like his brother is putting a wedge between us? What do I do?