Even with my limited experience as someone planning not to send my kids to school, I can tell you with some confidence that the most common objection people have to this idea is regarding socialization. As in:
How will your kids get socialized?
I used to be one of these people, so I am not here to pass judgment. I used to have the same concern. Now I realize that for me, it was a pretty automatic thing that popped into my head like a reflex when I heard the word "homeschool." I don't remember being taught to have this concern, but it got in my head somehow. Maybe because people told me that was why I was in school? I don't know.
I like to give people the benefit of the doubt, so I am going to assume that almost everyone who voices this concern is legitimately worried about my children. Worried about how they will learn to socialize without school. Worried that my children will grow up to be weird, or mean, or too shy or awkward, or I don't know what else.
So here are some questions for you about socialization. Think about them. Think about the reflex answers that pop into your head for some of them, and then really think about how you feel. If you want to. I would love to hear your answers if you are inclined to share. Use the "Anonymous" option if that helps.