There are two theories:
1) Babydoll accidentally kills her. The extended cut of the movie shows Babydoll shooting twice at her stepfather: First bullet misses his head, hitting the bulb instead, this bullet is not the killer bullet because its trajectory is too high to hit the little girl or the pipes. Second bullet misses the man's chest, hits his left forearm, finally hits one of the pipes hitting the little sister in between. The common cut of the movie shows her shooting only once and placing the gas escaping from the pipe on that scene making it not only confusing but impossible.
2) Some people believe Babydoll did not killed her sister but her stepfather. Babydoll then regrets for not being on time to save her and for causing him to go for the little one when fighting him back.
Two things in favor of theory 1 are: 1- when he kicks the door in, the little girl heads up and we can see the pipes behind her, she was on the right spot to die; 2- the narrative of the movie is more how the girls perceive/imagine things rather than how they exactly happened, so Babydoll could remember/imagine he kicked in before he really did; .
Two things in favor of theory 2 are: 1- when he already opened the closet she was still hanging outside, not giving her enough time to get in again, get the gun and reach them; 2- when she reaches there, he was facing downward, maybe at the already dead/hurt girl.
A big con of theory 2 is if he killed her, then Babydoll had no reason to spare him from death, instead it seems she realizes she is the one who killed her sister and not him.